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Abstract 


This  report  compiles  the  information  contained  in  three  Technical  Memoranda  pertaining  to  the 
development  of  Measures  of  Performance  (MOP)  for  Multi-Sensor  Data  Fusion  technology 
(MSDF),  and  evaluates  logistics  and  the  utility  of  the  Operations  Room  Team  Trainer  (ORTT)  to 
gather  these  MOPs.  Each  Technical  Memorandum  is  preceded  by  a  summary  of  its  content.  This 
report  recounts  the  development  of  a  strategy  for  collection  of  MOPs,  the  conduct  of  a  proof-of- 
concept  trial  at  ORTT  (with  attendant  identification  of  potential  improvements  to  the  data 
collection  strategy),  and  the  investigation  of  methods  to  reduce  the  logistical  and  availability 
problems  associated  with  obtaining  access  to  Subject  Matter  Experts  (SME).  As  a  result  of  this 
work,  a  further,  full-scale,  investigation  of  the  archived  ORTT  performance  data  to  provide 
performance  data  for  the  COMDAT  Technology  Demonstration  Programme  was  recommended. 
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Resume 


Le  present  rapport  comprend  une  compilation  de  l'information  contenue  dans  trois  documents 
techniques  portant  sur  l'elaboration  de  mesures  du  rendement  de  la  technologie  de  la  fusion  de 
donnees  multicapteurs  (FDMC),  ainsi  qu'une  evaluation  de  la  logistique  et  de  l'utilite  du  simulateur 
pour  l'equipe  de  la  salle  des  operations  (SESO)  en  vue  de  la  collecte  des  mesures  du  rendement. 
Chaque  document  technique  est  precede  d'un  resume  de  son  contenu.  Le  present  rapport  recapitule 
l'elaboration  d'une  strategic  de  collecte  des  mesures  du  rendement,  la  conduite  de  l'essai  de 
validation  de  principe  au  SESO  (Toperateur  identifiant  les  ameliorations  possibles  a  apporter  a  la 
strategic  de  collecte  des  donnees)  et  l'examen  des  methodes  visant  une  reduction  des  problemes  de 
logistique  et  de  disponibilite  associes  a  l'acces  aux  experts  en  la  matiere  (EM).  Suite  a  ce  travail,  un 
autre  examen  pleine  echelle  des  donnees  archivees  sur  le  rendement  du  SESO  a  ete  recommande  en 
vue  de  la  foumiture  de  donnees  sur  le  rendement  dans  le  cadre  du  programme  de  demonstration  de 
la  technologie  d'aide  aux  decisions  de  commandement  (COMDAT). 
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Executive  Summary 


Over  the  past  several  years,  Humansystems  Incorporated®  (HSI  ),  under  a  succession  of  contracts 
managed  by  DRDC-Toronto,  has  identified  the  critical  functions  and  their  associated  human  behaviours 
that  are  the  core  components  of  effective  Command  and  Control  (C2)  perfonnance  in  the  Operations 
Room  of  the  Halifax  Class  frigate.  The  work  commenced  with  a  Cognitive  Task  Analysis  (CTA)  of  the 
Operations  Room  Officer  (ORO),  using  a  scenario  based  methodology,  to  determine  the  major  functions 
performed  by  the  ORO.  The  analysis  focussed  on  situation  awareness,  communication,  decision 
making  and  workload.  Subsequently,  with  the  emergence  of  the  COMDAT  TD,  HSI  was  tasked  to 
develop  appropriate  Measures  of  Performance  (MOPS)  based  upon  the  CTA  that  would  be  appropriate 
for  assessing  the  impact  of  the  TD  on  operational  perfonnance.  In  addition,  HSI  was  requested  to 
develop  a  Test  and  Evaluation  (T&E)  Trial  Plan  and  to  evaluate  sites  and  opportunities  where 
appropriate  performance  data  might  be  collected. 

Several  independent  reports  and  technical  memoranda  resulted  from  this  work,  and,  although  complete 
in  themselves,  they  do  not  provide  a  comprehensive  view  of  the  work  that  has  been  done.  Therefore, 
DRDC-Toronto  has  requested  that  the  reports  be  organised  into  two  collections:  one  based  largely  on 
studies  relating  to  the  NCOT  facility,  the  other  relating  to  the  Operations  Room  Team  Trainer  (ORTT). 

Three  documents  are  first  summarised  and  then  presented  in  full  in  this  composite  report: 

The  first  report:  “ Technical  Memorandum :  Follow-Up  Evaluation  of  the  Operations  Room  Team 
Trainer  as  a  Test  Environment  for  Collecting  MOP  Data  to  Support  COMDAT ’  provides  an  overall 
context  for  the  remaining  reports.  It  deals  with  an  alternative  method  for  gathering  MOP  data  to  support 
the  COMDAT  Technology  Demonstration  Project.  This  method  involves  analysing  archived 
performance  data,  requiring  less  logistical  coordination  and  control  over  the  ORTT.  This  Technical 
Memorandum  reports  the  initial  assessment  of  using  ORTT  training  records  as  a  potential  source  of 
COMDAT  relevant  MOPs,  based  on  the  observation  of  an  actual  team  training  exercise  and  evaluation 
of  the  real-time  data  record. 

The  second  report:  “ Technical  Memorandum:  Assessing  the  Impact  of  Multi-Source  Data  Fusion  on 
Command  and  Control  Operations  in  the  Halifax-Class  Frigate:  Use  of  the  Operations  Room  Team 
Trainer  (ORTT)  Training  Records  to  Extract  Quantitative  and  Diagnostic  Measures  of  Performance” 
reports  on  a  detailed  proof  of  concept  evaluation  of  the  ability  to  extract  COMDAT  TD  relevant  MOPs 
from  archived  records  created  during  ongoing  Navy  training  scenarios  in  the  ORTT.  This  work  was 
expected  to  provide  a  definitive  direction  for  future  T&E  trials  concerning  the  use  of  either  NCOT  or 
ORTT  facilities. 

The  final  report  “ Technical  Memorandum:  Provision  of  Naval  SME  Support  to  Contractors  ”  addresses 
the  need  for  Subject  Matter  Experts  for  blocks  of  time  to  participate  in  trials  or  to  meet  with  contractors 
to  explain  operational  procedures  and  processes.  This  Technical  Memorandum  addresses  problems  in 
obtaining  this  access  and  seeks  to  find  a  way  to  lessen  demand  and  administrative  burden  whilst  still 
providing  the  required  SMEs  to  contractors. 

The  interested  reader  is  recommended  to  read  this  compilation  report  together  with  the  companion 
report  "Development  of  Measures  of  Performance  for  Evaluating  the  COMDAT  Technology 
Demonstrator:  Potential  Use  of  the  Naval  Combat  Operator  Trainer  (NCOT)  for  Data  Collection"  in 
order  to  be  fully  informed  regarding  the  development  of  MOPs  and  the  selection  of  appropriate  venues 
in  which  to  gather  MOPs  for  the  COMDAT  TD. 
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Sommaire 


Depuis  plusieurs  annees,  Humansystems  Incorporated®  (HSI  )  a,  en  vertu  d'une  serie  de  contrats 
administres  par  RDDC  Toronto,  identifie  les  fonctions  critiques  et  les  comportements  humains 
connexes  qui  constituent  les  elements  fondamentaux  du  commandement  et  du  controle  (C2)  efficaces 
dans  la  salle  des  operations  de  la  fregate  de  la  classe  Halifax.  Le  travail  a  commence  par  une  analyse 
cognitive  des  taches  de  l'officier  de  la  salle  des  operations,  au  moyen  d'une  methodologie  fondee  sur 
divers  scenarios,  dans  le  but  de  determiner  les  principales  fonctions  executees  par  l'officier  de  la  salle 
des  operations.  L'analyse  a  porte  principalement  sur  la  connaissance  de  la  situation,  la  communication, 
la  prise  de  decisions  et  la  charge  de  travail.  Par  la  suite,  avec  l'emergence  du  demonstrateur  de  la 
technologie  d'aide  aux  decisions  de  commandement  (COMDAT),  HSI  a  ete  chargee  d'elaborer  les 
mesures  du  rendement  appropriees,  d'apres  l'analyse  cognitive  des  taches  convenant  a  revaluation  de 
l'incidence  du  demonstrateur  de  technologie  sur  le  rendement  operationnel.  On  a  aussi  demande  a  HSI 
de  preparer  un  plan  d'essai  et  devaluation  (E  et  E)  et  d'evaluer  les  emplacements  et  les  occasions  ou  des 
donnees  appropriees  sur  le  rendement  pourraient  etre  recueillies. 

Plusieurs  rapports  et  documents  techniques  distincts  ont  ete  produits  au  terme  de  ces  travaux  et,  meme 
s'ils  sont  complets  en  soi,  ils  ne  donnent  pas  une  vue  d'ensemble  du  travail  effectue.  C'est  pourquoi 
RDDC  Toronto  a  demande  que  les  rapports  soient  regroupes  en  deux  series  :  une  serie  fondee 
principalement  sur  les  etudes  relatives  au  simulateur  d'operateur  d'equipement  de  combat  naval 
(NCOT),  et  l'autre  portant  sur  le  simulateur  pour  l'equipe  de  la  salle  des  operations  (SESO). 

Trois  documents  sont  tout  d'abord  resumes,  puis  presentes  dans  leur  version  integrate  dans  le  present 
rapport  global : 

Le  premier  rapport,  intitule  «  Technical  Memorandum'.  Follow-Up  Evaluation  of  the  Operations  Room 
Team  Trainer  as  a  Test  Environment  for  Collecting  MOP  Data  to  Support  COMDAT  »,  presente  le 
contexte  d'ensemble  des  autres  rapports.  II  porte  sur  une  autre  methode  de  collecte  de  donnees  de 
mesure  du  rendement  a  l'appui  du  projet  de  demonstration  de  la  technologie  COMDAT.  Cette  methode 
comprend  l'analyse  de  donnees  archivees  sur  le  rendement,  ce  qui  requiert  moins  de  controle  et  de 
coordination  logistiques  en  ce  qui  conceme  le  SESO.  Ce  document  technique  fait  etat  de  1'evaluation 
initiale  de  l'utilisation  des  dossiers  d'instruction  du  SESO  comme  source  potentielle  de  mesures 
pertinentes  du  rendement  de  la  technologie  COMDAT,  d'apres  l'observation  d'un  exercice  reel 
d'instruction  d'une  equipe  et  1'evaluation  du  dossier  de  donnees  en  temps  reel. 

Le  deuxieme  rapport,  intitule  «  Technical  Memorandum:  Assessing  the  Impact  of  Multi-Source  Data 
Fusion  on  Command  and  Control  Operations  in  the  Halifax-Class  Frigate:  Use  of  the  Operations  Room 
Team  Trainer  (ORTT)  Training  Records  to  Extract  Quantitative  and  Diagnostic  Measures  of 
Performance  »,  presente  une  evaluation  detaillee  de  validation  de  principe  de  la  capacite  a  extraire,  des 
dossiers  archives  crees  dans  le  SESO  durant  les  scenarios  d'instruction  en  cours  de  la  Marine,  les 
mesures  pertinentes  du  rendement  du  demonstrateur  de  la  technologie  COMDAT.  Ces  travaux  devaient 
permettre  de  donner  une  orientation  definitive  aux  luturs  E  et  E  concemant  l'utilisation  du  NCOT  ou  du 
SESO. 

Le  dernier  rapport,  intitule  «  Technical  Memorandum:  Provision  of  Naval  SME  Support  to 
Contractors  »,  porte  sur  la  necessity  de  retenir  les  services  d'experts  en  la  matiere  (EM)  pendant  des 
periodes  de  temps  en  vue  de  leur  participation  aux  essais  et  de  rencontres  avec  les  entrepeneurs  dans  le 
but  de  leur  expliquer  les  procedures  et  les  processus  operationnels.  Ce  document  technique  traite  des 
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difficultes  d'acces  aux  EM  et  cherche  a  trouver  un  moyen  d'amoindrir  la  demande  et  le  fardeau 
administratif,  tout  en  assurant  les  services  d'EM. 

On  recommande  au  lecteur  interesse  de  lire  le  present  rapport  de  compilation  et  le  rapport  connexe, 
intitule  «  Development  of  Measures  of  Performance  for  Evaluating  the  COMDAT  Technology 
Demonstrator:  Potential  Use  of  the  Naval  Combat  Operator  Trainer  (NCOT)  for  Data  Collection  », 
pour  obtenir  une  information  complete  sur  l'elaboration  des  mesures  du  rendement  et  la  selection  des 
emplacements  appropries  en  vue  de  la  collecte  des  mesures  du  rendement  du  demonstrateur  de  la 
technologie  COMDAT. 


Human. systems®  Incorporated 


COMDAT:  MOP  and  ORTT 


Page  vii 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


Page  viii 


COMDAT:  MOP  and  ORTT 


H  ummsystems  ®  Incorporated 


Table  of  Contents 


ABSTRACT . Ill 

RESUME  . IV 

EXECUTIVE  SUMMARY . V 

SOMMAIRE . VI 

TABLE  OF  CONTENTS . IX 

LIST  OF  TABLES . XI 

ACKNOWLEDGEMENTS . XII 

1.  INTRODUCTION . 1 

2.  REPORT  ORGANIZATION  AND  SUMMARY  OF  CONTENTS . 3 

3.  TECHNICAL  MEMORANDUM: . 5 

3.1  Summary . 5 

3.1.1  Background . 5 

3.1.2  Findings . 5 

3.1.3  Conclusions . 6 

3.2  Background . 7 

3 . 3  Goals  of  the  visit . 7 

3.4  Visit  Details . 8 

3.5  Analysis . 8 

3.5. 1  Characteristics  of  personnel  assessed  in  the  ORTT . 8 

3.5.2  Characteristics  of  the  training  scenario . 9 

3. 5. 3  Scenario  Record  and  Playback . 10 

3. 5. 4  Analysis  of  recorded  scenarios . 11 

3.5.5  Other  Issues . 12 

3.6  Discussion  and  Analysis . 12 

3.6.1  General  observations . 12 

3.6.2  Specific  points  of  comparison  between  NCOT  and  ORTT . 13 

3 . 7  Summary  and  Recommendations . 17 

3.8  References . 18 

4.  TECHNICAL  MEMORANDUM: . 19 

4.1  Summary . 19 

4.1.1  Background . 19 

4.1.2  Findings . 19 

4.1.3  Conclusions . 21 

4.2  Background . 22 

4 . 3  Goals  of  the  ORTT  trial . 24 

4.4  Method . 25 

4.4.1  Participants . 25 

4.4.2  General  Scenario . 26 

4.4.3  Equipment . 27 

4.4.4  Procedure . 29 

4.5  Results . 31 

4. 5. 1  Air  event  series:  description  and  overview . 32 

4.5.2  Sub-surface  event  series:  description  and  overview . 43 


blumansystems'  Incorporated  COMDAT :  MOP  and  ORTT  Page  ix 


4.5.3  Surface  event  series:  description  and  overview . 49 

4.5.4  Summary  of  MOPs  that  may  be  extracted  from  the  ORTT  data  record . 53 

4. 5. 5  Recommendations  for  subset  of  MOPs  to  be  used  for  assessing  COMDAT  MSDF  in  upcoming 

trials . 58 

4. 5. 6  The  efficiency  of  the  data  extraction  process . 59 

4.6  Discussion . 60 

4. 6. 1  Review  of  goals . 60 

4. 6. 2  General  assessment . 61 

4. 6.3  Data  collection  to  support  future  MSDF  TD  trials . 62 

4 . 7  Recommendations . 63 

4.8  References . 63 

5.  TECHNICAL  MEMORANDUM: . 65 

5.1  Summary . 65 

5.1.1  Background . 65 

5.1.2  Findings . 65 

5.1.3  Conclusions . 66 

5.2  Introduction . 66 

5.2. 1  Definition  of  the  Problem . 66 

5.2.2  The  Need  For  SMEs . 67 

5 . 3  Potential  Approaches . 67 

5.3. 1  Current  Structure . 67 

5.3.2  Centralized  SME  Availability’  Pool . 68 

5.3.3  Serving  SMEs  Attached  to  a  Project . 69 

5.3. 4  Serving  SMEs  Paid  in  their  Off  Time  as  Consultants . 69 

5.3. 5  Independent  SME  Pool  of  Navy’  Retirees . 70 

5.3. 6  Hybrid  Approach . 71 

5 . 4  Recommended  Approach . 71 

6.  OVERALL  SUMMARY . 74 

7.  LIST  OP  ACRONYMS . 76 

ANNEX  A:  ABILITY  OP  THE  ORTT  TO  SUPPORT  T&E  MOPS . 1 

ANNEX  B:  FEASIBILITY  AND  METHODS  FOR  PROVIDING  INFORMATION  TO  ORO  AND 
FOR  CAPTURING  MOPS . 1 

ANNEX  C:  SCENARIO  DATA  RECORD . 1 


ANNEX  D:  ELEMENTS  OF  AN  INDEPENDENT  SME  POOL  OF  NAVAL  RETIREES  FOR  THE 


COMDAT  PROJECT . 1 

Function:  Radar  Tracking/Above  Water  Picture  Compilation . 1 

Under  Water  Picture  Compilation . 2 

Electronic  Support  Measures  (ESM) . 2 

Fire  Control  (FC) . 3 

Aircraft  Control . 3 

Warfare  Sub-team  Direction . 4 

Ops  Room  Direction . 4 

Command . 5 

Tabular  Summary . 6 


Page  x 


COMDAT:  MOP  and  ORTT 


Ybxmwasystems  ®  Incorporated 


List  of  Tables 


Table  3.1:  Comparison  of  the  main  features  of  NCOT  and  ORTT  scenarios . 10 

Table  4.1:  Number  of  MOPs  derived  for  each  operational  area . 21 

Table  4.2:  Internal  and  external  communication  networks  organized  by  Ops  Room  position.  28 

Table  4.3:  Examples  of  ORO  centred  MOPs  collected  during  Pilot  Trial . 40 

Table  4.4:  Examples  of  Detect  to  Resolve  MOPs  (air  contacts)  collected  during  Pilot  Trial  42 

Table  4.5:  Examples  of  Detect  to  Resolve  MOPs  (sub-surface  contacts)  collected  during 

Pilot  Trial . 48 

Table  4.6:  Examples  of  Detect  to  Resolve  MOPs  (surface  contacts)  collected  during  Pilot 

Trial . 53 

Table  4.7:  Summary  of  MOPs  potentially  collectible  in  the  ORTT  based  upon  pilot  trial . 54 

Table  4.8:  Recommended  MOPs  for  assessing  MSDF  impact . 58 

T able  A 1 :  Summary  of  feasibility  of  collecting  MOP  data  in  the  ORTT . 4 

Table  B 1 :  Methods  assessed  in  ORTT  for  capturing  data  for  MOPs . 1 

Table  B2:  Scenario  events  and  ORO  actions  assessed  for  capturing  MOP  data . 2 


List  of  Figures 


Figure  4. 1 :  sector  assignments  and  disposition  of  the  Task  Group . 27 

Figure  4.2:  Gantt  chart  showing  main  events  and  actions  for  air  contacts . 33 

Figure  4.3 :  ORO  CCS  Display  at  Scenario  Start . 35 

Figure  4.4:  Gantt  chart  showing  main  events  and  actions  for  subsurface  contacts . 46 

Figure  4.5:  Screen  capture  of  ASPO  (RT2)  CCS  display  at  10: 17  in  scenario . 47 


Yinmansystems®  Incorporated 


COMDAT:  MOP  and  ORTT 


Page  xi 


Acknowledgements 


We  would  like  to  thank  the  following  individuals  and  organisations  for  making  the  pilot  trial 
possible. 

Lockheed  Martin  Canada 

Barry  Colboume,  ORTT  Technical  Representative,  Lockheed  Martin  Canada 
John  Milan,  ORTT  Manager,  Lockheed  Martin  Canada 
Cdr  Gord  Switzer,  Commandant  CFNOS 

LCdr  Darren  Gamier,  CFNOS  Tactics  and  Team  Training  Officer 


Page  xii 


COMDAT:  MOP  and  ORTT 


H um'&nsystems  ®  Incorporated 


1 


Introduction 


Over  the  past  several  years,  Humansystems  Incorporated  (HSI),  under  a  succession  of  contracts 
managed  by  DRDC-Toronto,  has  identified  the  critical  functions  and  their  associated  human 
behaviours  that  are  the  core  components  of  effective  Command  and  Control  (C2)  performance  in 
the  Operations  Room  of  the  Halifax  Class  frigate.  The  work  commenced  with  a  Cognitive  Task 
Analysis  (CTA)  of  the  Operations  Room  Officer  (ORO),  using  a  scenario  based  methodology,  to 
determine  the  major  functions  performed  by  the  ORO.  The  analysis  focussed  on  situation 
awareness,  communication,  decision  making  and  workload.  Subsequently,  with  the  emergence  of 
the  COMDAT  TD,  HSI  was  tasked  to  develop  appropriate  Measures  of  Performance  (MOPS) 
based  upon  the  CTA  that  would  be  appropriate  for  assessing  the  impact  of  the  TD  on  operational 
performance.  In  addition,  HSI  was  requested  to  develop  a  Test  and  Evaluation  (T&E)  Trial  Plan 
and  to  evaluate  sites  and  opportunities  where  appropriate  performance  data  might  be  collected. 

Several  independent  reports  and  technical  memoranda  resulted  from  this  work,  and,  although 
complete  in  themselves,  they  do  not  provide  a  comprehensive  view  of  the  work  that  has  been  done. 
Therefore,  DRDC-Toronto  has  requested  that  the  reports  be  organised  into  two  collections:  one 
based  largely  on  studies  relating  to  the  Naval  Combat  Operators  Trainer  (NCOT)  facility,  the  other 
relating  to  the  Operations  Room  Team  Trainer  (ORTT).  Thus,  in  order  to  obtain  a  comprehensive 
and  integrated  perspective  on  the  work  that  has  been  accomplished,  the  reader  is  encouraged  to 
review  both  reports. 
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2.  Report  organization  and  summary  of 
contents 


The  main  body  of  this  report  provides  a  summary  of  each  of  the  previously  independent  reports, 
followed  by  each  report  itself.  Where  necessary,  and  for  clarity,  linking  paragraphs  between 
reports  have  been  provided.  An  overall  summary  of  the  reports  is  provided  at  the  end  of  the 
document. 

The  first  report:  “ Technical  Memorandum :  Follow-Up  Evaluation  of  the  Operations  Room  Team 
Trainer  as  a  Test  Environment  for  Collecting  MOP  Data  to  Support  COM  DAT'  provides  an 
overall  context  for  the  remaining  reports.  It  deals  with  an  alternative  method  for  gathering  MOP 
data  to  support  the  COMDAT  Technology  Demonstration  Project.  This  method  involves  analysing 
archived  performance  data,  requiring  less  logistical  coordination  and  control  over  the  ORTT.  This 
Technical  Memorandum  reports  the  initial  assessment  of  using  ORTT  training  records  as  a 
potential  source  of  COMDAT  relevant  MOPs,  based  on  the  observation  of  an  actual  team  training 
exercise  and  evaluation  of  the  real-time  data  record. 

The  second  report:  “ Technical  Memorandum:  Assessing  the  Impact  of  Multi-Source  Data  Fusion 
on  Command  and  Control  Operations  in  the  Halifax-Class  Frigate:  Use  of  the  Operations  Room 
Team  Trainer  (ORTT)  Training  Records  to  Extract  Quantitative  and  Diagnostic  Measures  of 
Performance”  reports  on  work  whose  goal  was  to  conduct  a  detailed  proof  of  concept  evaluation  of 
the  ability  to  extract  COMDAT  TD  relevant  MOPs  from  archived  records  created  during  ongoing 
Navy  training  scenarios  in  the  ORTT.  This  work  was  expected  to  provide  a  definitive  direction  for 
future  T&E  trials  concerning  the  use  of  either  NCOT  or  ORTT  facilities. 

The  final  report  “ Technical  Memorandum:  Provision  of  Naval  SME  Support  to  Contractors  ” 
addresses  the  need  for  Subject  Matter  Experts  for  blocks  of  time  to  participate  in  trials  or  to  meet 
with  contractors  to  explain  operational  procedures  and  processes.  This  Technical  Memorandum 
addresses  problems  in  obtaining  this  access  and  seeks  to  find  a  way  to  lessen  demand  and 
administrative  burden  whilst  still  providing  the  required  SMEs  to  contractors. 
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3.  Technical  Memorandum: 


FOLLOW-UP  EVALUATION  OF  THE  OPERATIONS  ROOM  TEAM 
TRAINER  AS  A  TEST  ENVIRONMENT  FOR  COLLECTING  MOP  DATA 
TO  SUPPORT  COMDAT 

3.1  Summary 

3.1.1  Background 

The  original  assessment  of  the  ORTT  as  an  environment  in  which  to  collect  MOPs  for  Ops  Room 
functions  was  based  upon  the  premise  that  specialised  T&E  trials  would  be  the  means  by  which 
data  would  be  collected.  The  large  logistical  overhead  and  support  from  the  Navy  required  for 
mounting  such  a  trial  in  the  ORTT  was  considered  to  be  a  major  practical  barrier.  Therefore, 
attention  was  focussed  primarily  on  NCOT  as  a  facility  to  collect  small  team  MOPs.  A  number  of 
investigations  relating  to  the  feasibility  of  using  NCOT  are  documented  in  a  parallel  report 
“ Development  of  Measures  of  Performance  for  Evaluating  the  COMDAT  Technology > 
Demonstrator:  Potential  Use  of  the  Naval  Combat  Operator  Trainer  (NCOT)  for  Data 
Collection  As  a  result  of  the  NCOT  work,  several  technical  limitations  concerning  NCOT  data 
collection  and  analysis,  together  with  the  artificiality1  of  the  context  for  evaluating  COMDAT 
related  Ops  Room  team  performance,  led  to  a  re-consideration  of  the  ORTT  for  data  collection. 
However,  instead  of  viewing  the  ORTT  as  environment  for  a  dedicated  T&E  trial  as  originally 
conceived,  the  idea  of  extracting  MOPs  from  archived  ORTT  records  captured  during  sophisticated 
Navy  training  scenarios  was  thought  to  be  a  promising  avenue  of  approach.  Such  training 
scenarios  involve  intact  Ops  Room  teams,  with  suitable  levels  of  training  and  experience  and 
provide  a  high-fidelity  experience  involving  multi-warfare  threats  in  a  Canadian  Task  group/littoral 
water  environment. 

This  Technical  Memorandum  reports  the  initial  assessment  of  using  ORTT  training  records  as  a 
potential;  source  of  COMDAT  relevant  MOPs,  based  on  the  observation  of  an  actual  team  training 
exercise  and  evaluation  of  the  real-time  data  record. 

3.1.2  Findings 

The  observation  team  from  HS1 "  identified  nine  goals  for  their  visit  to  ORTT.  They  were: 

•  Observe  a  team  training  exercise; 

•  Evaluate  the  opportunities  for  data  capture; 

•  Assess  the  suitability  of  existing  ORTT  training  scenarios; 

•  Assess  the  playback  ability  of  ORTT; 


1  Teams  are  separated  from  the  normal  Ops  Room  and  Task  Group  wide  interactions  that  form  a  critical 
component  of  mission  execution  and  the  level  of  communications  is  artificially  low. 
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•  Evaluate  the  suitability  of  ORTT  for  T&E  data  collection; 

•  Determine  whether  MOP  data  can  be  extracted  from  archived  data; 

•  Assess  the  subject  populations  for  their  suitability  for  T&E  purposes; 

•  Assess  the  ability  of  ORTT  to  support  T&E  trial  logistics;  and 

•  Make  suggestions  for  future  test  trials. 

There  were  a  number  of  features  observed  in  the  ORTT  that  could  impact  future  attempts  to  use  it 
for  collection  of  MOPs.  One  of  the  most  important  was  the  requirement  to  use  existing  ORTT 
exercise  scenarios.  This  could  potentially  limit  the  project’s  ability  to  target  COMDAT  TD 
specific  behaviours  for  analysis.  Also,  any  exact  repetition  of  specific  scenario  threat  events  for 
comparative  purposes  could  be  difficult  to  achieve  due  to  the  real-time  reactive  actions  of  game 
controllers. 

A  detailed  feasibility  assessment  was  conducted  of  which  MOPs  could  be  collected  in  the  ORTT 
(based  upon  the  MOPs  evaluated  initially  in  NCOT)  and  what  aspects  of  the  COMDAT  TD  would 
require  the  development  of  new  MOPs.  An  additional  four  MOPs  were  created  to  meet  this  need, 
in  particular  for  the  context  of  ‘double-tracking’  (a  recurring  operational  problem  for  which  MSDF 
is  a  potential  solution). 

It  was  determined  that  ORTT  imposes  more  realistic  workload  on  the  ORO,  than  could  be  created 
in  NCOT  and  the  level  and  content  of  communications  between  team  members  were  also  more 
representative  of  the  operational  context.  Scenario  development  for  the  ORTT  for  T&E  purposes 
would  not  require  T&E  team  resources,  given  the  existence  of  robust  training  scenarios.  One  major 
advantage  of  the  ORTT  was  that  the  typical  scope  of  scenarios  involved  full  co-operation  among 
the  Ops  room  team  and  other  TG  units  and  there  were  many  repetitions  of  the  detect-to-resolve 
cycle. 

Data  recording  in  ORTT  was  also  found  to  be  more  comprehensive,  robust  and  reliable  than  in 
NCOT.  Further  data  playback  capabilities  in  the  ORTT  for  the  purposes  of  analysis  and  MOP 
extraction  represented  a  major  improvement  over  NCOT.  Further,  logistical  support  in  the  ORTT 
for  T&E  purposes  is  more  comprehensive  and  cheaper,  assuming  MOP  data  collection  is  integrated 
with  existing  training  exercises. 

There  were  some  potential  concerns  that  the  ORTT  might  not  be  able  to  provide  enough 
experimental  control  over  the  scenario  events.  Flence,  it  was  recommended  that  a  pilot  trial  be 
conducted  to  address  this  and  other  practical  issues. 

There  had  also  been  some  concern  that  security  requirements  associated  with  ORTT  might  make 
the  use  of  the  playback  facility  for  analysis  difficult,  but  as  a  result  of  the  initial  assessment,  it  was 
now  thought  that  this  would  not  be  a  significant  barrier,  providing  the  initial  analysis  and  data 
reduction  were  done  on  site  at  the  ORTT. 

3.1.3  Conclusions 

Based  on  the  factors  reported  above,  it  was  recommended  that  work  directed  at  the  collection  of 
MOPs  using  NCOT  be  ceased,  and  effort  be  directed  toward  the  conduct  of  a  pilot  trial  using  the 
ORTT  to  assess  the  suitability  of  using  archived  training  records  as  a  means  of  generating 
COMDAT  TD  relevant,  MOP  data. 
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3.2  Background 

In  the  past  year  we  reviewed  alternative  test  environments  including  Naval  Combat  Operator 
Trainer  (NCOT)  and  the  Operations  Room  Team  Trainer  (ORTT)  for  their  suitability  to  peimit  data 
collection  for  test  and  evaluation  purposes  (T&E)  in  support  of  the  COMDAT  program  (Matthews, 
Webb,  Stager  and  Keeble,  2001).  This  review  concluded  that  the  NCOT  would  be  the  most 
suitable  facility  for  conducting  T&E  using  single  operators  performing  simple  operational 
functions  such  as  detecting  and  tracking  radar  contacts.  While  more  limited  in  its  simulation 
fidelity  than  ORTT,  for  repeated  trials  of  simple  tasks,  the  NCOT  facility  appears  to  be  the  most 
versatile  to  configure  and  the  most  economical  to  operate.  NCOT  allows  the  collection  of  T&E 
data  for  several  operators  performing  the  same  task  in  parallel.  NCOT  also  appears  suitable  for 
assessing  performance  of  small  teams  of  2-4  individuals  performing  more  complex,  interacting 
functions.  This  assessment  was  based  upon  the  expectation  that  the  contractor  responsible  for 
NCOT  would  deliver  software  that  supports  this  level  of  functionality  within  the  next  year. 

At  the  time  of  the  original  evaluation,  we  did  not  have  an  opportunity  in  the  ORTT  to  observe  a  full 
exercise  in  progress  or  to  access  any  archived  data.  Nevertheless,  we  recognised  its  potential  for 
the  assessment  and  measurement  of  more  complex  functions  in  the  Ops  Room,  including  tasks 
involving  interactions  between  complete  warfare  teams,  interactions  between  warfare  directors  and 
the  ORO,  and  between  the  ORO,  CO  and  the  entire  Ops  Room  team. 

Since  that  time  we  have  conducted  a  proof  of  concept  evaluation  of  NCOT  (Matthews,  Webb  and 
Keeble,  2002)  in  which  we  noted  several  technical  limitations  with  the  scenario  creation  and  data 
playback  capabilities  of  NCOT,  which  still  remain  to  be  resolved.  Further,  in  contrast  to  previous 
expectations,  no  new  scenarios  had  been  developed  for  NCOT  that  are  readily  suitable  for  T&E 
purposes. 

In  parallel  to  the  ongoing  evaluation  of  NCOT,  and  following  discussions  with  the  Scientific 
Authority,  it  was  decided  to  further  investigate  the  capabilities  of  ORTT  particularly  with  respect  to 
the  possibility  of  being  able  to  extract  T&E  data  from  ongoing  Navy  training  exercises,  which  use 
the  facility  on  a  regular  basis.  It  was  further  felt  useful  to  visit  the  ORTT  during  an  actual  exercise 
to  observe  at  first  hand  a  functionally-operating  Ops  Room  team,  since  prior  analyses  and 
conclusions  had  been  largely  based  upon  reviews  of  technical  documentation,  discussions  with 
ORTT  and  NCOT  support  personnel  and  tabletop  walkthroughs  of  scenarios  with  Ops  Room 
SMEs. 

Consequently,  a  visit  was  arranged  for  a  1  luman.vi’.vte/u.v  Inc.  team  (M.  Matthews,  R.  Keeble,  L. 
Bruyn)  to  attend  the  ORTT  on  September  13-14,  2002. 

3.3  Goals  of  the  visit 

•  Observe  Ops  Room  team  functions  during  a  live  training  exercise 

•  Evaluate  opportunities  for  data  capture  during  exercise 

•  Examine  the  suitability  of  existing  scenarios  used  for  training  exercises 

•  Evaluate  the  ability  of  the  ORTT  for  data  replay  to  support  requirements  for  T&E 
analysis 

•  Re-visit  issues  on  suitability  of  the  ORTT  for  T&E  data  collection  during  training 
exercises 
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•  Determine  whether  MOP  data  could  be  extracted  from  archived  ORTT  exercise 
training  data. 

•  Assess  the  subject  populations  evaluated  in  the  ORTT  for  their  suitability  for  T&E 
purposes 

•  Determine  the  ability  of  the  ORTT  to  support  T&E  trial  logistics 

•  Recommend  courses  of  action  for  future  test  trials 


3.4  Visit  Details 

On  the  first  day  of  the  visit,  we  observed  a  full  Ops  Room  team  comprised  of  students  reaching  the 
end  of  their  the  various  CFNOS  operations  courses  and  who  were  in  final  preparation  for 
evaluation.  The  team  comprised  the  full  complement  of  personnel  that  are  normally  found  in  the 
ORTT.  Some  training  and  support  personnel  were  on  hand  to  intervene  with  the  team  should 
circumstances  warrant. 

We  were  able  to  observe  the  team  in  two  ways.  First:  by  positioning  ourselves  towards  the  side 
and  rear  in  the  Ops  Room  itself  adjacent  to  the  area  occupied  by  the  CO  and  communications 
station.  From  this  location  we  could  clearly  observe  most  of  the  activity  in  the  Ops  Room  without 
necessarily  being  able  to  view  the  content  of  CCS  displays.  Second,  we  re-located  to  the  adjacent 
Brief/Debrief  suite  where  we  were  able  to  monitor  up  to  three  workstations  at  a  time  and  listen  to 
one  Comm  net.  Other  workstations  and  Comm  nets  could  be  readily  selected  as  required.  Thus,  on 
the  first  day  we  looked  at  a  complete  training  scenario  run-through  lasting  about  2  hours.  We  also 
spent  time  in  becoming  familiar  with  the  replay  software  and  video  recording  capabilities. 

We  did  not  ask  for,  nor  have  access  to,  the  gaming  script  and  complete  gaming  package  provided  to 
the  exercise  team  including  information  such  as  rules  of  engagement  (ROE)  and  identification 
criteria  etc,  although  we  believe  this  could  be  made  available  if  requested. 

The  entire  second  day  was  spent  in  the  Brief/Debrief  suite  where  we  reviewed  and  analysed  a 
previously  recorded  training  scenario  that  used  the  same  scenario  events  that  we  had  observed,  but 
involved  a  different  team.  Assisting  us  in  the  use  of  the  facility  were  the  ORTT  manager  and 
another  EMC  staff  member. 

3.5  Analysis 

3.5.1  Characteristics  of  personnel  assessed  in  the  ORTT 

Essentially  three  types  of  personnel  come  to  the  ORTT  for  training  and  assessment.  The  first  group 
comprises  students  nearing  the  end  of  a  particular  operations  Qualification  Level  (QL)  course 
ranging  from  QL  5  to  ORO.  They  spend  approximately  seven  developmental  days  in  the  trainer 
going  through  a  series  of  scenarios,  which  culminates  with  an  individual  assessment  of  their 
abilities  to  meet  the  course  training  standards.  This  group  of  individuals  would  not  have  worked 
together  as  a  team  before  and  all  are  approximately  equally  inexperienced  in  the  roles  for  which 
they  are  working  to  qualify. 

The  second  group  of  trainees  involves  individuals  who  are  qualified  for  their  Ops  Room  position 
but  have  been  away  from  operational  deployment.  For  these  individuals,  the  ORTT  represents  a 
refresher  course. 
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The  third  group  comprises  a  fully  functioning  Ops  Room  team  actively  deployed  in  a  ship  who 
have  come  to  the  trainer  to  gain  additional  practice  and/or  familiarity  with  specific  or  new 
procedures.  Examples  of  this  would  be  preparations  for  a  live  missile  shoot,  or  for  a  deployment  to 
an  operational  theatre. 

Clearly,  the  first  group  will  have  the  most  technical  proficiency  in  the  basic  procedures  but 
virtually  no  experience  in  applying  those  procedures  at  sea.  The  second  group  will  be  somewhat 
stale  in  both  their  technical  and  operational  proficiency.  The  third  group  will  have  more 
operational  experience,  but  depending  upon  these  experiences  and  habits  formed,  may  have  less 
technical  proficiency  in  some  aspects  of  using  the  Ops  Room  equipment  than  the  first  group.  This 
group  will  also  have  developed  a  more  complete  team  mental  model  than  the  other  two  groups, 
which  will  impact  both  upon  their  intra-team  communication  and  the  manner  in  which  they 
collectively  problem  solve. 

From  a  future  T&E  perspective,  the  optimum  subject  sample  would  comprise  individuals  from 
group  3.  Flowever,  the  opportunity  to  compare  across  teams  in  the  various  groups  should  not  be 
overlooked,  since  this  would  provide  potentially  valuable  data  on  the  completeness  of  the  different 
aspects  of  their  team  mental  models,  including  knowledge  of  tasks,  knowledge  of  other  team 
members,  knowledge  of  the  system  and  knowledge  of  context. 

3.5.2  Characteristics  of  the  training  scenario 

In  general,  open  water  scenarios  are  used  for  team  training,  although  a  littoral  scenario  is  currently 
under  development.  The  scenarios  involve  a  typical  Canadian  Navy  task  group  with  three  or  more 
Halifax  class  frigates,  one  mission  essential  unit2,  an  Iroquois  class  destroyer  and  a  maritime  patrol 
aircraft  (MPA).  Scenario  background  events  comprise  surface  marine  traffic  and  limited 
commercial  flights  along  high  level  air  corridors.  Threat  events  involve  attacks  by  aircraft  and 
ships  firing  anti-ship  missiles  and  submarines  firing  torpedoes. 

A  typical  2-hour  scenario  usually  has  the  following  elements  and  comprises  three  45-minute 
segments  that  focus  on  air  (AAW),  surface  (ASuW)  and  underwater  (UWW),  culminating  with  a 
simultaneous  threat  in  two  warfare  areas: 

•  5-6  underwater  contacts:  (torpedo,  whales,  wrecks,  decoys,  friendly  &  threat  sub) 

•  30  surface  contacts 

•  Air  contacts:  approx.  1  every  6  minutes  mostly  commercial  air,  no  neutral  helos,  or 
neutral  air 

In  addition,  the  scenario  support  team  can  manufacture  entities  as  in  real  time  as  well  as  load  mini¬ 
scenarios  within  each  scenario  (all  pre-scripted).  A  description  of  the  planned  events  for  the 
scenario  is  available,  to  which  we  can  have  access. 

The  following  table  shows  some  of  the  major  differences  in  scenario  between  that  typically  used  in 
the  ORTT  for  training  at  that  developed  by  HSI  for  possible  use  in  T&E  in  NCOT. 


ORTT  Scenario 

HSI/NCOT  Scenario 

Open  water 

Littoral 

Low  volume,  high  altitude  commercial  air 

Moderate-high  volume,  high  altitude  commercial  air 

2  Formerly  called  "high  value  unit" 
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No  low  altitude  commercial  air 

Moderate-high  volume,  low  altitude  commercial  air 
(helos  a/c  landing  and  departing  from  nearby  airports) 

Low  volume,  slow  moving  surface  traffic 

Higher  volume,  slow  and  fast  surface  traffic 

Sub-surface  contacts 

None 

Repeated  air  threats 

Limited  number  of  air  threats,  many  potential  air  threat 
situations 

Several  air-launched  anti-ship  missile  attacks 

None 

Torpedo  attack 

None 

Engagement  of  incoming  missiles 

None 

Engagement  of  attacking  a/c 

None 

Engagement  of  surface  ships 

None 

Avoidance  of  incoming  torpedo 

None 

Double  tracking  can  occur  when  track  is  held  by 
two  sensor  systems 

Could  not  successfully  simulate 

Script  is  modified  in  real  time  by  game  players  to 
reflect  circumstances 

Scripted  events  follow  a  more  precise  schedule. 

Table  3.1:  Comparison  of  the  main  features  of  NCOT  and  ORTT  scenarios 

ORTT  scenario  limitations: 

•  It  is  not  possible  to  create  a  scenario  that  exactly  matches  all  T&E  requirements  since 
ORTT  pre-scripted  scenarios  must  be  used. 

•  The  scenario  will  not  be  exactly  repeatable  across  different  test  runs  and  teams, 
because  the  game  controllers  take  into  account  ongoing  circumstances  and  how  the 
team  is  performing  when  introducing  threat  events. 

3.5.3  Scenario  Record  and  Playback 

The  ORTT  functionality  allows  all  Ops  Room  CCS  displays  and  communications  to  be  recorded 
from  the  entire  team  present.  Face  to  face  interactions  and  the  passage  of  paper  communications 
(largely  between  the  ORS  and  other  team  members)  can  be  recorded  through  a  separate  video  and 
audio  monitoring  system,  using  one  camera  and  two  microphones  to  capture  the  entire  Ops  Room. 
Since  we  judged  the  current  placement  of  these  to  be  less  than  optimum  for  capturing  ORS/ORO 
exchanges,  the  ORTT  manager  was  willing  to  explore  the  possibility  of  relocating  these  on  a 
temporary  basis  for  any  future  T&E  trials  in  the  ORTT. 

The  scenario  playback  can  be  monitored  in  the  Brief/Debrief  room  either  on  three  wall  mounted 
large  screen  displays  or  15"  CRT  monitors  (note:  larger  monitors  could  be  made  available  to  the 
test  team  for  analysis/playback  purposes).  The  data  replay  must  be  done  through  the  hardware  at 
ORTT. 

The  ORTT  scenario  playback  function  has  the  following  capabilities: 

•  Can  play  at  half,  full  and  5  X  speed 

•  Can  set  flags  during  scenario  execution 
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•  Can  tag  points  in  time  during  playback  and  make  notes  for  quick  location  during 
playbacks 

•  Can  pause  and  resume  at  any  time 

•  Can  jump  playback  to  a  specific  time  or  to  a  marker 

•  Can  select  any  three  of  all  the  Ops  Room  CCS  workstations  for  playback 

•  Can  listen  to  one  audio  circuit/net  at  one  time  (user  selectable) 

The  latter  difficulty  may  be  overcome  in  analysis  by  repeated  runs  through  critical  parts  of  the 
playback  and  listening  to  different  circuits.  Alternately,  all  of  the  existing  soundtracks  could  be 
multiplexed  onto  a  tape  (during  recording)  for  subsequent  simultaneous  replay. 

In  addition  to  the  above,  the  separate  ORTT  video  and  audio  record  can  also  be  replayed  on  a 
separate  large  screen  independently  of,  or  in  synchrony  with,  the  CCS/Comm  digital  data  record. 

3.5.4  Analysis  of  recorded  scenarios 

As  noted  earlier,  analysis  of  scenarios  would  have  to  be  conducted  at  the  ORTT  Brief/Debrief 
room.  The  ORTT  manager  does  not  believe  that  there  would  be  any  problem  in  getting  blocks  of 
access  time  in  this  room,  since  it  is  not  widely  used  by  the  Navy. 

In  general,  the  analysis  will  be  conducted  through  repeated  runs  of  the  scenario  record 
supplemented  by  separate  audio/video  record.  In  order  to  document  and  measure  the  information 
flow  and  team  actions,  we  believe  that  it  would  be  beneficial  to  develop  a  Gantt-chart  format 
showing  the  detailed  relationship  of  scenario  events  and  team  actions  and  communications  on  a 
common  time  line. 

One  important  issue  that  arises  from  our  present  review  of  the  ORTT,  is  how  we  would  be  able  to 
time  events  of  interest  with  the  required  precision,  given  that  the  time  line  of  the  scenario  is  not 
scripted  to  the  second.  An  example  of  this  would  be  determining  the  start  of  each  detect-to-resolve 
cycle3.  In  general,  many  initial  contacts  will  come  from  MPA  and  will  therefore  first  appear  on 
screen  as  linked  tracks;  other  contacts  will  come  from  auto-initiated  radar  tracks.  We  believe  that  if 
we  have  the  script  in  hand  and  can  be  present  in  the  game  control  facility  during  scenario 
execution,  we  can  readily  observe  these  events  and  place  a  T&E  marker  on  the  scenario  record  at 
the  commencement  of  selected  detect  to  resolve  cycles.  Thus,  during  playback  we  will  be  able  to 
determine  the  first  point  in  time  at  which  some  form  of  response  or  action  on  any  track  could  have 
been  initiated.  Determining  the  timing  of  these  responses  during  playback  will  not  be  a  problem  as 
both  the  CCS  time  clock  and  scenario  replay  clocks  are  readily  visible.  Even  if  we  cannot  be 
present  during  scenario  execution  and  have  to  rely  solely  on  determining  the  start  of  some  events 
from  the  playback  record,  by  using  cues  in  the  script  and  closely  monitoring  the  appropriate  CCS 
display  where  an  event  is  about  to  occur,  we  should  be  able  to  see  the  time  of  first  occurrence  on 
the  display. 

Analysis  of  non-CCS  and  non-net-based  communications  would  come  primarily  from  the  separate 
audio-video  tape  record,  which  would  require  manual  synchronisation  with  the  ORTT  digital 
record. 


3  The  detect-to-resolve  cycle  is  considered  a  critical  task  in  evaluating  MSDF  or  other  decision  aid 
technologies  that  are  aimed  at  improving  the  tactical  picture  and  reducing  response  time  to  potential  threats. 
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A  detailed  analysis  (similar  to  the  one  conducted  for  the  NCOT  POC)  of  how  the  measurement  of 
specific  MOPs  would  be  conducted  and  is  supported  by  the  ORTT  is  provided  in  Annexes  A  and  B. 

3.5.5  Other  Issues 

3. 5. 5.1  Threat  responses 

Because  the  ORTT  scenarios  always  require  the  team  to  respond  to  threats,  it  would  be  relatively 
straightforward  to  incorporate  MOPs  for  this  into  the  analysis  process. 

3. 5. 5. 2  Dual  tracking  by  radar 

Dual  tracking  by  radar  is  considered  a  primary  problem  for  which  MSDF  technology  should 
provide  a  significant  benefit.  Prior  to  the  ORTT  visit  we  have  not  been  able  to  observe  how  Halifax 
class  radars  actually  behave  in  conditions  of  signal  uncertainty  and  we  have  not  been  in  a  position 
to  prescribe  MOPs  for  dual  tracking  situations.  Having  now  seen  how  and  when  these 
circumstances  occur,  we  recommend  the  following  additional  MOPs: 

•  Frequency  of  dual  track  situations  (system-wide  measures  rather  than  ORO  measure) 

•  Time  to  detect  and  respond  to  dual  track  situation 

•  Total  time  spent  in  dealing  with  dual  track  situation 

•  Accuracy  in  responding  to  dual  track  situation  (e.g.  do  they  put  STIRS  on  the 
appropriate  track) 

3. 5. 5. 3  Preferred  subject  population 

For  formal  testing,  we  would  prefer  to  have  ship’s  team  coming  directly  from  sea  because  it  would 
be  more  likely  to  represent  the  skill  level  to  be  expected  in  actual  operational  performance.  This 
type  of  team  would  be  in  the  second  phase  of  training  as  opposed  to  initial  training  and  would  have 
developed  the  critical  experience  in  working  together  on  common  procedures. 

However,  at  some  future  time  it  would  be  useful  to  collect  data  on  all  three  groups  to  allow  the 
assessment  of  how  COMDAT  may  impact  upon  Ops  Room  performance  with  teams  at  varying 
stages  of  training  and  experience  and  hence  different  team  mental  models. 

3.6  Discussion  and  Analysis 

3.6.1  General  observations 

Our  first  and  most  important  observation  having  closely  watched  an  Ops  Room  team  at  work  for 
the  first  time  at  any  length  and  in  any  detail,  is  that  the  work  and  performance  of  the  ORO  is 
inextricably  bound  with  the  team  around  him/her.  This  clearly  reinforces  the  conclusion  from  the 
cognitive  task  analysis  (CTA)  of  the  ORO  (Matthews,  Webb  and  Bryant,  1999).  Many  tasks 
involve  a  close  synchrony  of  actions,  communications  and  common  understandings  between  the 
ORO  and  team  members,  in  particular  the  CO,  SWC,  ASWC  and  ORS.  In  turn,  the  reactions  and 
behaviour  of  the  warfare  directors  are  in  turn  integrated  with  their  respective  teams.  Thus,  attempts 
to  measure  the  performance  of  ORO  based  tasks  in  isolation  from  the  team  could  represent  a  major 
challenge  to  the  construct  validity  of  such  measures.  Not  only  are  many  of  the  OROs  tasks 
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embedded  within  the  team  functions  and  actions,  but  the  performance  is  highly  dependent  in  many 
cases  on  the  specific  actions  and  effectiveness  of  the  other  team  members.  Therefore,  to  isolate  the 
ORO  and  assess  him/her  in  an  environment  that  examines  solely  his/her  own  tasks  in  the  context  of 
an  unknown  and  unpractised  team  (as  had  been  contemplated  for  NCOT)  may  be  problematic.  The 
ensuing  data  collected  from  such  an  approach  may  achieve  reliability  and  robustness  yet  it  may 
lack  ecological  validity  and  representativeness  of  the  true  operational  context. 

In  comparing  the  NCOT  and  ORTT  as  potential  test  sites  in  our  previous  report,  we  had 
recommended  NCOT  for  assessment  of  tasks  involving  operators  and  small  teams  and  the  ORTT 
for  the  assessment  of  more  complex  teams  and  the  full  Ops  Room.  Having  now  seen  how  even 
small  sub-teams  perform  in  the  wider  context  of  ongoing  Ops  Room  functions,  we  feel  that  this 
initial  assessment  may  need  to  be  qualified.  Given  the  ORO's  involvement  with  the  broader  Ops 
Room  functions,  it  does  not  appear  that  an  NCOT  configuration  of  an  air  warfare  team  (or  other 
domain)  plus  the  ORO  can  be  justified  when  the  measurement  level  of  interest  is  the  ORO.  NCOT 
would  be  more  suitable  for  measures  addressed  at  the  operator  or  warfare  director  level.  Further, 
while  the  workload  for  such  a  team  may  be  satisfactorily  simulated  in  NCOT,  the  workload  for  the 
ORO  is  unrepresentative,  except  for  those  situations  where  operational  circumstances  require  the 
ORO  to  perform  the  functions  of  the  SWC.  As  we  noted  from  the  CTA,  the  ORO  is  primarily  a 
multi-tasker,  one  moment  a  manager,  another  a  co-ordinator,  another  a  facilitator,  another  a 
planner,  another  an  analyser,  another  a  decision  maker  etc.  To  remove  the  ORO  from  the  context 
of  such  workload  and  attempting  to  evaluate  his/her  performance  in  an  isolated  sub-team,  as  had 
been  attempted  and  planned  for  NCOT,  severely  misrepresents  the  operational  circumstances  under 
which  the  ORO  performs.  As  such,  this  has  the  danger  of  both  limiting  the  generalisability  and 
validity  of  any  performance  measures  obtained  in  this  way. 

3.6.2  Specific  points  of  comparison  between  NCOT  and  ORTT 

We  will  now  look  in  more  detail  at  some  of  the  advantages  and  disadvantages  of  each  environment 
with  respect  to  the  major  needs  for  developing  and  conducting  T&E  trials 

3. 6. 2.1  Scenario  development 

The  development  of  scenarios  in  NCOT  for  MOP  purposes  will  require  the  HSI  team  to  write  and 
code  the  entire  scenario,  since  none  exist  at  present. 

For  the  ORTT,  several  blue  water  scenarios  are  in  place  and  have  been  validated  through  a  number 
of  training  exercises.  Littoral  scenarios  are  under  development. 

Conclusion :  use  of  the  ORTT  will  save  considerable  resources  and  associated  costs  in  scenario 
development  and  will  provide  validated  scenarios. 

3. 6. 2. 2.  Scenario  scope 

In  NCOT,  the  custom  written  scenario  that  HSI  has  developed  is  based  upon  a  littoral  environment 
and  reflects  a  relatively  low  threat  to  background  event  pattern.  The  scenario  also  involves  air  and 
surface  contacts.  The  Ops  Room  team  spends  much  of  its  time  dealing  with  background  traffic  and 
the  scenario  may  be  fairly  representative  of  such  operational  conditions.  The  scenario  does  not 
involve  any  direct  weapons  threats  to  the  ship  and  requires  no  action  to  be  taken  against  potential 
threats.  Radar  tracking  problems,  of  the  type  thought  to  be  of  interest  to  MSDF,  are  difficult  to 
simulate  in  NCOT. 
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The  ORTT  scenario  focuses  more  on  the  detection  and  response  to  threats  in  all  warfare  areas 
against  a  lower  volume  of  background  events.  The  scenario  escalates  to  the  point  where  tactical 
decisions  concerning  threat  priority  are  forced  upon  the  Ops  Room  team.  Dual  radar  tracking  of 
targets  occurs4  readily  and  represents  opportunities  to  document  and  measure  how  well  MSDF 
technology  improves  Ops  Room  performance. 

Conclusion :  the  HSI  scenario,  while  operationally  realistic,  provides  fewer  opportunities  for 
observing  the  detect-to-resolve-to-react  cycle  across  all  warfare  domains.  Since  the  focus  of 
COMDAT  MSDF  technology  is  to  improve  the  tactical  picture,  particularly  to  resolve  problems 
with  threat  identification  and  tracking,  the  ORTT  scenario  provides  a  suitable  environment  for 
collecting  appropriate  MOP  data. 

3. 6.2.3  T&E  measurement  needs 

These  are  outlined  in  detail  in  Annexes  A  and  B  in  the  same  format  as  the  NCOT  POC  report.  The 
major  difference  between  the  two  environments  concerns  the  control  afforded  the  T&E  team  in 
NCOT  compared  with  the  ORTT.  In  NCOT  the  T&E  team  would  be  able  to  inject  probes,  stop  the 
scenario,  ask  for  SITREPS,  communicate  with  players  and  control  events  through  the  use  of  "actors" 
in  the  warfare  teams.  None  of  these  capabilities  can  be  achieved  directly  in  the  ORTT. 

Conclusion :  The  degree  to  which  the  lack  of  "experimental"  control  in  the  ORTT  represents  a 
barrier  to  the  collection  of  the  range  of  MOP  data  required  remains  to  be  documented.  The  original 
T&E  plan  represented  a  desire  to  have  an  optimum  T&E  environment,  the  question  is  whether  the 
ORTT  will  be  an  adequate  environment  that  satisfices  T&E  needs.  It  is  recommended  that  this  be 
investigated  through  a  pilot  test  to  determine  what  useful  MOP  data  can  be  extracted  from  archived 
exercise  data. 

3. 6. 2. 4  Scenario  execution 

In  NCOT  the  scenario  is  executed  in  the  same  way  every  time  according  to  the  programmed  schedule 
of  events.  This  has  the  advantage  of  repeatability  and  experimental  control,  but  it  remains  to  be 
determined  whether  this  can  be  adequately  co-ordinated  with  ship  movements  over  a  two-hour 
scenario,  or  whether  some  T&E  team  intervention  will  be  required.  Such  intervention  requires  expert 
knowledge  in  how  to  realistically  place/time  the  event  in  question.  In  the  ORTT,  the  entity 
controllers  follow  a  general  script  of  events  but  vary  their  timing  and  place  of  entry  into  the  scenario 
according  to  current  circumstances  of  the  mission  and  the  ongoing  workload  of  the  team. 

Conclusion:  The  reduced  ability  to  precisely  control  scenario  events  and  to  repeat  scenarios 
exactly  across  trials  may  be  a  necessary  trade-off  against  other  advantages  of  the  ORTT. 

3. 6.2. 5  Team  communication 

Aside  from  the  use  of  a  real  (ORTT)  as  opposed  to  a  simulated  SHINCOM  (NCOT)  panel,  the 
ORTT  provides  the  full  range  of  internal  and  external  communications  that  would  be  very  difficult 


4  We  believe  that  the  dual  tracking  that  we  witnessed  was  specific  to  tracks  within  the  quick-reaction  (QR) 
range,  and  had  more  to  do  with  extra  track  generation  than  some  of  the  dual  tracking  situations  experienced 
at  sea,  where  radars  may  have  difficulty  in  tracking  close  or  violently  manoeuvring  aircraft.  Based  upon  this 
single  scenario,  we  are  not  in  a  position  at  this  stage  to  comment  upon  whether  the  dual  tracking  in  the  ORTT 
would  occur  as  readily  as  it  would  at  sea. 
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to  duplicate  in  NCOT  without  the  use  of  an  extensive  support  team.  Further,  it  would  be  difficult  to 
achieve  the  volume  and  salience  of  text  messages  that  are  directed  to  the  ORO,  largely  through  the 
ORS.  Further,  our  observations  of  the  training  exercise  underlined  the  degree  to  which  in  actual 
operations  that  the  ORO  is  constantly  changing  his  communication  focus  and  interactions.  While 
the  NCOT  POC  showed  that  we  could  successfully  reproduce  the  communication  activity  of  a 
small  warfare  team,  this  severely  underestimates  the  typical  communication  load  for  the  ORO  in  a 
normal  Ops  Room.  Further,  we  observed  the  critical  role  played  by  the  ORS  in  supplementing  net- 
based  communications  in  the  Ops  Room.  Such  a  role  and  capability  would  be  difficult  to 
implement  in  NCOT. 

Conclusion :  The  ORTT  represents  a  more  realistic  and  valid  context  for  assessing  ORO 
communication  effectiveness  both  within  and  external  to  the  Ops  Room. 

3. 6. 2. 6  Da  ta  recording 

The  data  recording  capabilities  of  the  ORTT  and  NCOT  are  somewhat  similar,  although  the  ORTT 
is  specifically  designed  to  capture  Ops  Room  wide  data  in  an  integrated  manner.  To  date  it  has  not 
been  proven  that  NCOT  can  capture  as  broad  a  base  of  data,  and  there  have  been  indications  that 
data  storage  capacity  would  be  exceeded  if  data  were  to  be  captured  for  anything  larger  than  small 
teams.  In  the  ORTT,  audio  data  are  captured  from  each  of  the  separate  comm  nets  and  stored 
independently  for  each  net,  whereas  in  NCOT  audio  data  are  captured  on  a  workstation  basis.  The 
ORTT  provides  a  capability  for  the  placement  of  flags  or  markers  of  points  of  interest  for  T&E 
purposes  (from  a  workstation  in  the  control  room)  into  the  captured  record,  which  will  assist  in  the 
subsequent  analysis  of  data.  Such  markers  can  be  given  a  descriptor  tag  at  the  time  of  creation. 
NCOT  provides  no  similar  equivalent  capability  but  would  have  to  be  worked  around  by  having  a 
member  of  the  T&E  team  hit  a  spare  key  or  function  button  on  one  of  the  workstations  and  the  time 
of  such  action  would  have  to  be  manually  logged. 

Conclusion :  For  Ops  Room  wide  functions  and  team  interactions,  the  ORTT  provides  a  more 
comprehensive,  robust  and  reliable  environment  for  data  capture.  The  ORTT  provides  better 
support  for  adding  T&E  flags  to  facilitate  the  analysis  the  captured  record. 

3. 6. 2. 7  Support  for  da  ta  analysis 

In  NCOT  the  data  record  for  each,  separate  workstation  can  only  be  played  back  on  separate, 
individual  workstations;  synchrony  of  the  playback  of  the  data  records  from  multiple  workstations 
requires  intensive  manual  control5.  While  this  may  be  achievable  for  small  teams,  it  would  be 
impractical  for  Ops  room  wide  playback.  In  the  ORTT,  all  of  the  workstation  data  are  captured  in 
an  integrated  manner  and  the  system  is  designed  to  replay  selected  workstations  in  synchrony.  One 
limitation  of  the  ORTT  is  that  only  three  workstations  at  any  one  time  can  be  selected  for  review, 
necessitating  repeated  runs  through  the  data  record  in  order  to  build  the  Ops  room  wide  picture  (but 
this  would  also  be  a  practical  limitation  in  NCOT). 

In  NCOT  the  user  control  over  playback  allows  for  normal  play  speed,  fast  forward  to  a  designated 
time  in  the  record  and  pause.  After  the  playback  record  is  paused/stopped  it  automatically  goes  back 
to  the  start  of  the  entire  record,  with  no  option  to  continue  from  the  pause  point.  In  the  ORTT,  the 
record  can  be  replayed  at  .5,  lx,  5x  normal  speed,  paused  and  resumed  and  can  be  advanced  to 


5  Or  would  require  the  use  of  ancillary  data  capture/playback  equipment  such  as  the  DGx  digital  recorder. 
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previously  set  flags  (during  scenario  execution).  The  user  can  also  tag  points  in  time  during  playback 
and  append  notes,  which  can  be  readily  accessed  during  subsequent  playbacks. 

In  the  ORTT,  unlike  NCOT,  audio  playback  is  tightly  synchronised  with  the  video  playback, 
thereby  allowing  all  net-based  communications  to  be  heard  in  context  of  what  is  happening  on  the 
tactical  display.  The  sound  quality  of  audio  playback  did  not  suffer  from  the  problems  of  noise 
intrusion  and  sometimes  poor  signal  to  noise  ratios  found  in  NCOT. 

Conclusion :  The  ORTT  provide  more  of  the  necessary  controls  over  replay  of  the  data  record  to 
support  the  extraction  of  information  and  data  for  T&E  data  analysis.  The  ORTT  provides  more 
accurate  synchronisation  of  the  record  and  audio  from  multiple  workstations. 

3. 6. 2. 8  Logistical  support  -  general 

For  NCOT,  the  T&E  team  would  be  required  to  allocate  significant  effort  and  resources  in  suitable 
scenario  creation,  whereas  in  the  ORTT  ready  made,  suitable  scenarios  are  available. 

For  NCOT,  additional  Navy  resources  would  be  required  to  support  each  trial  to  act  as  team 
members.  In  the  ORTT,  this  is  not  an  issue  as  Navy  personnel  are  receiving  training  for  all  team 
positions  and  data  capture  involves  no  further  navy  resources  to  act  as  team  members. 

In  the  ORTT,  each  scenario  is  executed  with  the  background  support  of  a  full  team  of  game 
controllers  and  players,  who  respond  to  (and  initiate)  all  team  communications,  control  game  piece 
movement  and  miscellaneous  scenario  events.  In  NCOT  such  roles  would  have  to  be  played  by 
T&E  and  Navy  personnel. 

For  the  ORTT,  LMC  support  would  be  required  during  scenario  analysis  only  and  for  providing 
access  to  the  control  room  during  an  exercise  run  (to  allow  the  T&E  team  to  set  flags  etc).  For 
NCOT  support  would  be  required  during  scenario  preparation,  execution  and  analysis. 

To  enhance  the  capabilities  of  the  NCOT  replay  to  meet  T&E  analysis  requirements,  additional 
software  would  need  to  be  created  and  the  purchase  of  a  digital  data  recording  system  would  be 
necessary  to  achieve  the  same  degree  of  CCS  data  reply  as  currently  available  in  the  ORTT  without 
any  software  modifications. 

Conclusion :  Personnel  costs  to  conduct  T&E  trials  are  significantly  lower  in  the  ORTT  than 
NCOT.  The  ORTT  comes  ready  to  use  without  requiring  additional  expenditures  on  software  and 
hardware.  Most  importantly,  by  taking  advantage  of  the  ongoing  Navy  training  cycles  in  the 
ORTT,  the  need  to  continually  request  sometimes  scarce  Navy  personnel  resources  to  assist  in  the 
conduct  of  T&E  trials  using  NCOT  would  be  eliminated. 

3. 6.2.9  A  vailability/access  of  facility 

In  general,  in  NCOT  our  requests  for  access  to  date  have  been  able  to  be  met  around  the  existing 
training  schedules.  For  the  ORTT,  Navy  utilisation  of  the  facility  appears  to  be  at  a  high  level. 
However,  if  we  are  only  interested  in  the  analysis  of  archived  scenario  data  this  can  be  done  without 
disruption  to  ongoing  training  in  the  simulated  Ops  Room,  since  analysis  is  conducted  in  the  adjacent 
Brief/Debrief  Room.  One  caveat  at  present  is  that  such  analysis  cannot  be  conducted  at  the  same  time 
as  scenario  execution,  because  there  is  no  available  software  to  support  playback  on  a  second  unit, 
which  currently  only  serves  as  a  backup  (but  could  be  made  available  for  playback).  Without  this 
alternate  playback  capability,  T&E  needs  would  have  to  be  met  in  evenings,  overnight  and  at 
weekends  with  some  associated  LMC  costs  to  support  the  T&E  team.  However,  the  ORTT  manager 
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believes  that  CFNOS  could  also  benefit  from  the  development  of  a  second  playback  capability,  as  it 
would  allow  instructors  to  debrief  some  students  while  others  use  the  ORTT  for  live  training.  His 
estimate  is  that  it  might  take  as  little  as  two  days  programming  to  change  the  configuration  to  allow 
dual  live  scenario  record  /  previous  scenario  playback. 

To  date,  the  Navy  has  been  willing  to  accommodate  the  T&E  requirement  for  T&E  personnel  to  be 
present  in  the  simulated  Ops  Room  or  control  facility. 

Conclusion :  Existing  limitations  on  access  to  the  ORTT  for  playback  would  not  be  a  significant 
barrier  to  the  conduct  of  some  T&E  trials  to  collect  pilot  data  in  the  near  future.  For  more 
extensive  analysis  of  a  larger  sample  of  scenarios,  some  arrangement  would  need  to  be  made  to 
contract  the  additional  software  revisions  for  independent  playback.  There  is  a  possibility  that  the 
Navy  may  be  willing  to  be  a  partner  in  such  costs. 

3.6.2. 10  Availability  of  test  participants 

For  each  T&E  session  in  NCOT,  the  T&E  team  would  have  to  request  the  Navy  for  test 
participants.  In  the  case  of  OROs,  as  was  mentioned  in  earlier  reports,  this  may  be  difficult  to 
achieve  given  the  sample  size  needs  and  the  availability  within  the  small  population  of  OROs. 

Further,  as  the  proof  of  concept  in  NCOT  showed,  the  experience  level  and  operational  familiarity 
with  personnel  provided  may  not  be  optimum.  On  the  other  hand,  for  the  ORTT,  Navy  personnel 
are  continually  undergoing  training,  with  only  a  4-6  week  break  annually  (usually  less).  As  a 
result,  the  ORTT  has  the  potential  for  allowing  a  significant  T&E  database  to  be  created  with  a 
broad  cross-section  of  Navy  training  and  experience  levels  represented. 

Conclusion:  The  ORTT  provides  the  potential  for  the  collection  of  a  more  extensive  and 
comprehensive  data  set  that  is  more  reliable  and  more  generalizable  to  the  population  under  study. 

3.7  Summary  and  Recommendations 

The  exploratory  visit  to  ORTT  has  provided  an  opportunity  to  conduct  a  quasi  proof-of-concept 
analysis  that  goes  beyond  the  initial  assessment  of  the  facility  and  parallels  the  recent  analysis 
performed  for  NCOT.  In  comparing  the  results  of  our  analysis  for  the  two  environments,  it  is  clear 
that  neither  represents  a  100%  solution  for  T&E  needs.  For  both  there  are  some  practical  and 
logistical  trade-offs  that  must  be  made.  In  general,  for  NCOT  we  have  a  higher  degree  of 
experimental  control  over  scenario  events,  can  conduct  probes,  pause  the  execution  and  control  the 
behaviour  of  team  members  not  under  study.  For  the  ORTT  we  have  less  of  such  control,  but  we 
achieve  lower  costs  in  trial  preparation,  execution  and  analysis,  and  place  less  demands  on  scarce 
navy  SME  resources.  Some  MOPs  already  considered  may  not  be  achievable  in  the  ORTT,  but 
opportunities  for  others  may  present  themselves.  Further,  the  sophisticated  playback  capabilities 
available  in  the  ORTT  will  allow  the  painstaking  task  of  extracting  performance  data  to  be  conducted 
with  a  much  higher  level  of  efficiency  than  in  NCOT  (assuming  that  the  replay  software  can  be  made 
to  work  in  the  latter  to  better  support  analysis  than  is  presently  the  case).  However,  the  overwhelming 
positive  advantage  of  the  ORTT  is  access  to  a  potentially  large  database  of  performance  data  that  is 
likely  to  be  more  representative  of  the  Navy.  This  will  result  in  a  higher  level  of  reliability  and 
external  validity  than  could  ever  probably  be  achieved  with  NCOT  given  our  likely  success  in  being 
able  to  get  access  to  scarce,  qualified  Navy  personnel. 

Therefore,  we  recommend  that  we  cease  current  efforts  to  bring  NCOT  to  an  improved  level  of 
capability  to  support  T&E  needs  (i.e.  supplementary  digital  data  recording,  software  revisions)  and 


Wummsystems"  Incorporated 


COMDAT:  MOP  and  ORTT 


Page  17 


divert  our  efforts  to  conducting  a  pilot  trial  in  the  ORTT.  The  goals  of  this  trial  will  be  to  develop 
methods  for  analysing  an  existing  scenario  record  and  to  yield  performance  data  on  a  selected 
subset  of  MOPs. 
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4.  Technical  Memorandum: 

ASSESSING  THE  IMPACT  OF  MULTI-SOURCE  DATA  FUSION  ON 
COMMAND  AND  CONTROL  OPERATIONS  IN  THE  HALIFAX  CLASS 
FRIGATE:  USE  OF  THE  OPERATIONS  ROOM  TEAM  TRAINER  (ORTT) 
TRAINING  RECORDS  TO  EXTRACT  QUANTITATIVE  AND  DIAGNOSTIC 
MEASURES  OF  PERFORMANCE 

4.1  Summary 

4.1.1  Background 

The  goal  of  this  project  was  to  conduct  a  detailed  proof  of  concept  evaluation  of  the  ability  to 
extract  COMDAT  TD  relevant  MOPs  from  archived  records  created  during  ongoing  Navy  training 
scenarios  in  the  ORTT.  It  was  expected  that  the  outcome  would  provide  a  definitive  direction  for 
future  T&E  trials  concerning  the  use  of  either  the  NCOT  or  ORTT  facilities. 

4.1.2  Findings 

The  advantages  of  using  archived  data  to  assess  the  impact  of  MSDF  technologies  include: 

•  No  additional  burden  on  the  Navy; 

•  Large  potential  subject  sample; 

•  Many  data  collection  opportunities  due  to  ongoing  training; 

•  Large  database  of  existing  records;  and 

•  Better  data  from  intact  teams. 

However,  some  disadvantages  do  exist,  including  a  lack  of  direct  T&E  team  control  over  the  trial, 
inconsistent  conduct  of  similar  or  same  training  scenarios,  and  no  possibility  of  requesting  a 
SITREP  during  the  actual  scenario  for  the  purposes  of  assessing  Situation  Awareness,  should  the 
need  arise. 

The  goals  of  this  Proof-Of-Concept  (POC)  trial  were: 

•  Observe  and  identify  potential  MOPs; 

•  Assess  the  practicality  of  inserting  tags  into  scenario  recordings; 

•  Develop  methods  for  recording  and  analysis  of  data; 

•  Analyse  the  video  and  audio  record  from  ORTT  software; 

•  Analyse  the  video  and  audio  record  from  supplemental  sources  (e.g.  video  camera); 

•  Derive  quantifiable  MOPs; 

•  Determine  the  range  of  MOPs  possible  with  this  method; 
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•  Recommend  a  subset  of  MOPs  most  suitable  for  assessing  the  COMDAT  TD;  and 

•  Comment  on  the  effectiveness  and  efficiency  of  data  extraction  and  the  analysis 
method. 

Results  of  the  analysis 
Sources  for  data  analysis  were: 

•  A  video  record  of  all  CCS  screen  events 

•  An  audio  record  of  all  net  based  communications 

•  A  data  file  containing  a  time-ordered  list  of  keystrokes  from  each  Ops  Room  console 

•  An  audio-video  record  of  team-interactions  captured  by  a  CCTV  camera  and 
microphones  during  the  scenario 

The  method  of  analysis  consisted  of  three  steps:  the  reconstruction  of  critical  events  using  the  CCS 
video  record  and  the  network  communication  record  along  a  common  time  base;  a  semantic 
analysis  of  the  output  from  the  first  step,  and  then  the  extraction  and  derivation  of  relevant  MOPs 
from  the  first  two  steps.  While  the  first  two  steps  needed  to  be  done  using  the  ORTT  playback 
capability,  the  last  step  could  be  done  off  site.  It  was  found  that  the  first  step  took  roughly  36 
person-hours  to  document  90  real-time  scenario  minutes.  The  second  and  third  steps  took  30 
person-hours  in  total. 

In  general,  the  CCS  video  record  was  found  to  be  easier  to  analyse  than  the  communications  record 
because  the  analysts  could  view  replay  from  three  Ops  Room  consoles  at  once,  whereas  only  one 
audio  net  could  be  played  back  at  a  time.  Further,  the  screen  data  was  less  transient  than  the  audio, 
which  made  it  easier  to  assess  the  critical  information  from  relevant  team  workstations  at  one  time. 
Much  of  the  time  in  analysis  was  taken  up  with  the  extraction  of  the  sequencing  of  communication 
messages,  the  identification  of  sources  and  recipients  and  the  piecemeal  reconstruction  of  the 
information  flow  among  the  team. 

The  data  recording  of  all  keystrokes  captured  from  each  CCS  console  was  found  to  take 
significantly  more  effort  to  analyse  compared  to  the  three  steps  described  above  and  was  not 
pursued  at  length.  It  was  established  that  there  could  also  be  security  implications  associated  with 
removing  the  raw  keystroke  data  from  the  ORTT  for  more  detailed  analysis  off-site. 

The  separate  audio  and  video  tape  records  of  inter-team  interactions  (not  CCS  or  comm,  based) 
were  only  needed  to  be  consulted  if  other  analyses  were  ambiguous,  or  data  were  missing  or 
corroboration  was  required. 

The  appropriate  COMDAT  TD  relevant  MOPs  derived  from  this  POC  were  a  subset  of  the  MOPs 
based  upon  the  original  ORO  CTA  and  first  examined  in  NCOT.  The  table  below  lists  the  number 
of  MOPs  associated  with  each  functional  operational  area  and  specific  COMDAT  relevant  tasks. 

In  total,  52  MOPs  were  derived. 
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0R0  MOPs  for  air  events 

27 

0R0  MOPs  for  sub-surface  events 

7 

0R0  MOPs  for  surface  events 

4 

Additional  team  MOPs 

14 

MOPs  specific  to  MSDF 

12 

Table  4.1 :  Number  of  MOPs  derived  for  each  operational  area 
4.1.3  Conclusions 

In  general,  all  of  the  trial  goals  were  achieved  satisfactorily 

It  was  concluded  that  in  comparison  with  NCOT,  ORTT  is  considered  a  better  environment  from 
which  to  collect  COMDAT  TD  relevant  MOP  data.  MOPs  for  a  range  of  critical  tasks  in  the  detect 
to  resolve  process  could  be  extracted  from  the  data  record  for  air,  surface  and  sub-surface  events. 

Five  recommendations  resulted  from  this  study: 

•  A  full  data  collection  trial  should  be  conducted  using  ORTT  training  records  to 
generate  the  baseline  comparative  MOP  data  set  (using  the  MOPs  short-listed  in  this 
report)  for  MSDF  relevant  contexts  and  tasks. 

•  A  standalone  debrief  facility  should  be  developed  that  can  be  used  while  ORTT  is 
running  to  permit  more  flexible  scheduling  of  data  analysis  and  more  efficient  data 
extraction. 

•  Assuming,  the  MSDF  TD  is  to  be  implemented  in  the  CSTC,  a  pilot  trial  in  this 
facility  should  be  conducted  to  fully  evaluate  the  ability  to  collect  the  MOPs  identified 
for  the  evaluation  of  the  impact  of  the  MSDF  technology. 

•  To  facilitate  the  collection  of  data  in  the  CSTC  (should  the  need  arise),  acquire,  or 
make  available,  supplementary  digital  data  capture,  such  as  the  DGx  system. 

•  Conduct  an  evaluation  of  the  capability  of  the  COMDAT  MSDF  prototype  workstation 
and  software  to  collect  MOP  data  for  test  and  evaluation  purposes. 
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4.2  Background 

This  report  represents  part  of  a  continuing  contribution  to  the  understanding  and  measurement  of 
processes  that  support  Command  and  Control  (C2)  in  the  Operations  Room  (Ops  Room)  of  the 
Halifax  Class  frigate.  This  program  of  work  has  been  contracted  to  HumanSystems  Incorporated 
(HSI®)  by  DRDC-Toronto  and  has  the  current  goal  of  developing  reliable,  valid  and  representative 
Measures  of  Performance  (MOPs)  of  Ops  Room  processes  that  can  be  realistically  collected  in  a 
comprehensive  Test  and  Evaluation  (T&E)  program. 

The  work  has  its  historical  roots  in  a  review  and  analysis  performed  by  HSI®  into  the  wide  range 
of  measures  that  have  been  used  to  assess  C2  and  other  military  processes.  This  review  resulted  in 
recommendations  for  how  formal  evaluation  may  be  conducted  for  generic  military  C2  contexts 
(Matthews,  Webb  and  McCann,  1997).  Subsequently,  the  focus  for  analysis  has  been  the  Ops 
Room  of  the  Halifax  Class  frigate,  where  a  number  of  related  studies  have  been  conducted  in  recent 
years.  The  most  relevant  to  the  current  report  are  a  Cognitive  Task  Analysis  (CTA)  of  the  Ops 
Room  Officer  (ORO)  (Matthews,  Webb  and  Bryant,  1999)  and  subsequent  validation  (Matthews 
and  Webb,  2000),  and  the  development  of  measures  and  methods  to  collect  MOP  data  to  assess 
Ops  Room  C2  performance,  with  a  focus  on  the  ORO  (Matthews,  Webb  &  Keeble,  2001). 

The  current  work  focus  is  in  direct  support  of  the  COMDAT  Technology  Demonstration  Project 
(TDP).  The  objective  is  to  develop  MOPs  that  will  be  useful  for  quantitatively  evaluating  the 
impact  of  new  technologies  resulting  from  this  program  on  human-system  operational  performance 
in  the  Ops  Room.  Further,  the  collection  of  baseline  MOPs  on  a  variety  of  Ops  Room  critical  tasks 
will  provide  an  important  point  of  reference  for  developing  and  evaluating  a  variety  of  future  Navy 
initiatives.  These  include  Ops  Room  redesign,  integration  of  new  systems,  changes  to  training  and 
new  training  technologies  and  the  analysis  and  development  of  tactics  and  Ops  Room  procedures. 

In  leading  up  to  the  present  work  a  number  of  preliminary  studies  have  been  conducted  as  part  of 
the  overall  T&E  Plan,  which  is  described  in  Matthews,  Webb  and  Keeble  (2001).  Initially,  as 
suggested  in  this  plan,  HSI®  conducted  a  Proof  of  Concept  (POC)  trial  in  the  Naval  Combat 
Operator  Trainer  (NCOT)  with  a  view  to  assessing  its  suitability  to  support  T&E  for  the  collection 
of  MOPs  for  a  small  Ops  Room  Anti-Air  Warfare  (AAW)  team.  Limitations  of  the  NCOT  facility 
revealed  as  a  result  of  the  POC  (Matthews,  Webb,  Keeble,  2002)  and  the  potential  difficulty  in 
obtaining  the  quantity  and  range  of  Navy  Subject  Matter  Experts  (SMEs)  to  support  T&E  in  this 
environment  have  suggested  a  re-focussing  of  future  data  collection  efforts  on  the  Operations 
Room  Team  Trainer  ORTT  facility. 

Initially,  the  T&E  plan  had  considered  adapting  the  ORTT  to  meet  the  specific  requirements  for 
conducting  T&E  trials,  whereby  the  ORTT  would  be  used  exclusively  to  collect  T&E  data  with  test 
participants  specifically  recruited  for  each  trial.  However,  difficulty  in  booking  time  at  the  facility 
for  T  &  E  purposes,  as  well  as  the  overhead  that  would  be  required  of  the  Navy  for  both  running 
the  ORTT  and  providing  test  participants  just  for  T&E  purposes,  suggested  consideration  of 
another  strategy  for  MOP  data  collection  in  the  ORTT.  Since  the  ORTT  is  continuously  used  for 
training  full  Ops  Room  teams  for  much  of  the  year,  with  a  large  range  of  trainee  levels,  the 
possibility  was  considered  of  extracting  MOP  data  captured  from  such  training  exercises  using  the 
built-in  ORTT  data  recording  and  playback. 

Such  an  approach  has  several  merits. 

•  No  additional  burden  on  the  Navy  for  T&E  trials  with  support  personnel  and  trial 
participants. 
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•  Data  collected  from  a  wide  range  of  Navy  personnel,  thereby  improving  reliability  and 
generalisability  of  MOP  data  collected. 

•  Ongoing  use  of  the  ORTT  for  training  provides  many  opportunities  for  collecting  T&E 
data. 

•  A  large  ORTT  database  of  exercise  records  representing  Navy  personnel  operating  under 
quasi-operational  conditions  from  which  performance  data  can  be  extracted. 

•  Detailed  data  from  T&E  assessments  using  intact  teams  could  provide  important  insights 
to  the  Navy  about  team  performance  issues  not  readily  ascertained  when  the  team  is 
assessed  during  training. 

Using  pre-recorded  scenario  data  from  the  ORTT  was  also  seen  to  have  some  disadvantages. 

•  No  direct  T&E  control  over  the  trial,  for  example  to  vary  circumstances  in  a  systematic 
manner  or  to  inject  situational  awareness  "probes"  into  the  information  stream. 

•  Scenarios  could  not  be  repeated  precisely  because  training  exercise  controllers 
introduce  events  depending  upon  how  the  team  under  observation  is  performing. 

•  T&E  could  not  request  a  "SITREP"  to  assess  situational  awareness  of  any  team 
member  at  a  particular  point  in  time. 

Despite  such  limitations,  on  balance,  scenarios  recorded  in  the  ORTT  for  Navy  training  are  a 
valuable  potential  source  for  the  collection  of  data  for  MOPs.  Accordingly,  a  feasibility 
assessment  was  conducted  at  ORTT  and  a  technical  memorandum  delivered  to  the  Scientific 
Authority  (Matthews,  Keeble  and  Bruyn,  2002).  The  conclusions  from  that  assessment  are 
summarised  below. 

Comparing  the  results  of  the  POC  for  the  NCOT  and  ORTT  environments,  neither  represents  a 
100%  solution  for  T&E  needs.  For  both  there  are  practical  and  logistical  trade-offs  that  must  be 
made. 

•  For  NCOT,  in  general,  there  is  a  higher  degree  of  experimental  control  over  scenario 
events,  an  ability  to  conduct  probes,  pause  the  execution  and  control  the  behaviour  of 
team  members  not  under  study  (assuming  the  ORO  is  the  point  of  focus). 

•  For  ORTT  there  is  less  control,  but  lower  costs  are  associated  with  trial  preparation, 
execution  and  analysis,  and  fewer  demands  are  placed  on  scarce  navy  SME  resources. 
While  some  MOPs  already  considered  as  part  of  the  T&E  plan  may  not  be  achievable 
in  the  ORTT,  the  more  sophisticated  playback  capabilities  in  ORTT  allow  greater 
efficiency  for  the  painstaking  task  of  extracting  performance  data  than  NCOT.  This  is 
so  even  if  NCOT  replay  software  can  be  improved. 

•  The  biggest  single  advantage  of  the  ORTT  is  the  large  database  of  exercise  records 
representing  Navy  personnel  operating  under  quasi-operational  conditions  from  which 
performance  data  can  be  extracted.  This  will  result  in  a  higher  level  of  reliability  and 
external  validity  than  could  ever  probably  be  achieved  with  NCOT,  even  if  given 
unlimited  access  to  scarce,  qualified  Navy  personnel  for  trials  in  NCOT. 

•  Finally,  in  ORTT,  realistic  workload  levels  are  much  easier  to  achieve,  especially  for 
the  ORO  and  other  team  leaders,  since  a  full  Ops  team  environment  can  be  provided 
for  the  team  leader(s)  to  have  something  to  do  other  than  merely  monitor  their  screens. 

Therefore,  on  the  basis  of  the  two  POC  trials,  we  recommended  to  the  Scientific  Authority  that  we 
cease  ongoing  efforts  to  bring  NCOT  to  an  improved  level  of  capability  to  support  T&E  needs  (i.e. 
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supplementary  digital  data  recording,  software  revisions).  Instead,  we  recommended  diverting  our 
efforts  to  conducting  a  pilot  trial  in  the  ORTT.  The  goals  of  this  trial  being  to  develop  methods  for 
analysing  an  existing  scenario  record  and  to  collect  sample  performance  data  on  a  selected  subset 
of  MOPs. 

The  Scientific  Authority  has  concurred  with  this  recommendation  and  agreed  to  conduct  a  small 
pilot  trial  in  the  ORTT  to  demonstrate  the  type  of  MOP  data  that  could  be  extracted  from  the 
scenario  record.  A  secondary  goal  was  to  develop  and  assess  the  specific  methodologies  and 
procedures  that  would  be  required  for  the  data  extraction.  The  balance  of  this  report  presents  the 
outcome  of  this  pilot  trial. 

4.3  Goals  of  the  ORTT  trial 

Before  outlining  the  specific  goals  of  the  trial,  we  will  briefly  review  the  overarching  goals  of  the 
T&E  program.  Two  types  of  goals  are  associated  with  this  ongoing  program,  one  more  science 
based,  the  other  more  practical  to  the  Navy  community,  but  both  are  related  and  have  common 
elements. 

The  science-based  goals  reflect  an  ongoing  analysis,  conceptualisation  and  measurement  of 
processes  in  Command  and  Control.  To  this  end,  we  have  identified  key  processes  and  functions 
performed  by  the  ORO  and  have  suggested  a  research  and  measurement  plan  to  further  the  goal. 
The  more  general  goals  of  this  program  are  to  look  at  decision  support  tools  for  the  Ops  Room  for  a 
wide  range  of  functions  that  go  beyond  picture  compilation  and  include  engagement  and  response 
functions  and  process  monitoring. 

For  COMDAT,  the  more  practical  goal  is  to  evaluate  the  operational  impact  of  MSDF  technology 
on  critical  Ops  Room  tasks.  This  technology  impacts  most  directly  on  the  picture  compilation 
component  of  the  overall  detect-to-engage  process6.  We  believe  that  the  technology  can  best  be 
assessed  by  measuring  key  processes  in  the  detect-to-resolve7  cycle  using  appropriate  MOPs. 

These  MOPs  are  expressed  in  quantitative  terms  such  as  percent  accuracy,  percent  errors,  time  to 
perform  certain  tasks,  total  time  spent  in  associated  communications  and  percent  communication 
errors. 

As  a  concrete  example,  let  us  consider  the  team  task  of  identifying  an  unknown  air  contact.  By 
collecting  baseline  performance  data  it  might  be  established  that  the  average  time  from  the  first 
appearance  of  an  unknown  contact  on  the  CCS  to  its  identification  (for  example)  is  7.5  sec  with  an 
accuracy  rate  of  98%  and  five  associated  communications  totalling  40  seconds.  These  baseline 
numbers  enable  evaluation  of  performance  on  similar  tasks  with  the  benefit  of  MSDF  technology 
and  permit  the  Navy  to  draw  informed,  data-based  conclusions  as  to  the  operational  impact  of  the 
proposed  technology. 

For  some  Ops  Room  functions  performance  cannot  be  reflected  by  observable  tasks  that  can  be 
timed  or  assessed  for  accuracy  (e.g.  wide  area  picture  integration,  tactical  assessment).  For  such 
cases,  we  propose  the  use  of  subject  matter  expert  (SME)  assessment  using  valid  and  reliable 


6  This  is  our  best  assessment  of  how  MSDF  technology  will  impact  on  Ops  Room  tasks  based  upon  our  understanding  of  the  current 
progress  of  the  TDP.  Should  MSDF  have  impacts  on  other  tasks  beyond  picture  compilation  and  the  detect-to-resolve  cycle,  then  a 
more  suitable  subset  of  MOPs  will  need  to  be  selected  from  the  complete  set  already  identified. 

7  By  resolve  we  mean  the  overt  act  of  assigning  an  ID  (Assume  Friend,  Friend,  Suspect,  Hostile)  to  a  contact.  This  should  not  be 
confused  with  the  formal  "resolve"  process,  which  the  SWC  initiates  with  “All  positions,  resolve  track  XXXX”. 
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behaviourally  anchored  rating  scales  to  compare  performance  of  individuals  and  teams  with  and 
without  the  benefit  of  MSDF  technology. 

For  the  purpose  of  the  present  trial  with  its  COMDAT  MSDF  focus,  we  have  concentrated  on  the 
detect-to-  resolve  process  and  surface,  air  and  sub-surface  picture  building.  These  detect-to-resolve 
processes  include  tasks  such  as  contact  detection  and  identification  and  threat  assessment  that  fit 
under  the  general  CTA  ORO  function  of  "Manage  Ship  Surveillance".  Flowever,  as  indicated  above, 
other  functions  and  MOPs  relating  to  threat  engagement  and  the  ORO's  monitoring  of  Ops  Room 
tasks  have  been  previously  identified  as  candidates  for  improved  decision  support  though  probably 
outside  the  influence  of  MSDF.  These  could  become  the  focus  of  data  collection  using  the  methods 
outlined  below,  should  future  needs  arise.  Thus,  a  full  range  of  potential  MOPs  to  support  these 
longer-term  goals  has  also  been  identified  in  the  pilot  trial. 

The  specific  goals  of  the  ORTT  POC  trial  were  as  follows: 

•  Observe  a  training  scenario  in  action  and  note  areas  of  interest  to  the  T&E  team  for 
potential  MOP  extraction. 

•  Assess  the  practicality  of  inserting  T&E  "tags"  into  the  scenario  record  during  scenario 
execution  to  assist  in  subsequent  MOP  analysis. 

•  Develop  a  methodology  for  analysis  and  recording  the  captured  data. 

•  Analyse  the  digital  video  and  audio  record  (using  ORTT  Debrief  software)  to  build  the 
ongoing  team  picture  and  communications  for  selected  events. 

•  Analyse  the  supplementary  video  and  audio  record  (using  a  standard  VCR)  to  build 
the  ongoing  team  picture  and  communications  for  selected  events. 

•  Derive  quantifiable  MOPs  from  the  previous  analyses. 

•  Determine  range  of  potential  MOPs  that  could  be  assessed  using  the  methodology 

•  Make  recommendations  for  the  specific  subset  of  MOPs  that  would  be  most  suitable 
for  assessing  the  impact  of  MSDF  technology  in  future  trials 

•  Comment  upon  the  effectiveness  and  efficiency  of  the  data  extraction  and  analysis 
methodology  and  its  implications  for  logistical  and  other  support  in  future  trials. 


4.4  Method 

The  trial  was  conducted  during  an  existing  ORTT  training  exercise  in  which  a  full  ship's  team  is 
being  assessed.  The  ORTT  consists  of  two  F1FX  Ops  room  simulators  (Cubicles  1  and  2)  each  with 
an  associated  bridge  simulator,  thereby  permitting  simulator  training  exercises  involving  two 
complete  ships  Ops  room  plus  bridge  teams.  In  this  case,  one  team  was  being  evaluated,  while  the 
other  team  was  not,  and  acted  under  the  directions  of  exercise  control  to  assist  with  the  evaluation. 

4.4.1  Participants 

The  personnel  in  the  ORTT  trainer  during  the  pilot  study  comprised  two  Ops  Room  teams,  one  in 
each  cubicle  of  the  trainer.  The  team  in  cubicle  1,  which  we  will  refer  to  as  Team  1,  was  a  fully 
functioning  Ops  Room  team  from  a  Halifax  class  ship  which  had  recently  returned  from 
operational  deployment,  and  was  going  through  work-ups.  This  team  had  been  in  the  ORTT  trainer 
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for  the  entire  week  (i.e.  four  days  prior  to  the  pilot  study)  and  was  being  evaluated  on  the  day  of  the 
pilot  study.  In  the  four  days  prior  to  the  pilot  study,  the  team  had  worked  through  a  variety  of 
basic,  predefined  scenarios,  which  enabled  the  team  to  practice  their  teamwork  skills  prior  to 
evaluation  on  the  final  day.  The  team  in  cubicle  2  (from  the  same  ship),  which  we  will  refer  to  as 
Team  2,  represented  another  fully  functioning  Ops  Room  team  but  was  not  being  evaluated  at  the 
time  of  the  pilot  study.  We  limited  our  data  collection  to  Team  1,  on  the  assumption  that  because 
they  were  being  evaluated,  the  team  member’s  performance  would  be  more  representative  of 
typical  Ops  Room  team  performance  on  an  operational  mission. 

4.4.2  General  Scenario 

This  scenario  description  is  for  illustration  only  to  provide  a  context  for  the  events  that  have  been 
recorded.  It  is  not  intended  to  represent  all  aspects  of  the  mission  scenario  that  would  be  detailed  in 
the  OPGEN  message. 

A  Canadian  Task  group  is  operating  in  open  ocean  blue  water.  It  has  the  standard  composition  of: 

•  One  replenishment  ship  (Preserver) 

•  One  Iroquois  class  destroyer  (Algonquin) 

•  Three  Halifax  class  frigates  (Halifax,  Vancouver,  Regina) 

•  Four  embarked  Sea  King  helicopters  (2  active  dipping  CH  124A,  2  passive  CH  124B) 

•  One  MPA  (British  Nimrod) 

There  is  a  known  friendly  submarine  in  the  vicinity,  but  there  appears  to  be  no  other  friendly  task 
group.  Two  friendly  Combat  Air  Patrol  (CAP)  will  join  the  TG.  There  are  high  level,  commercial 
air  lanes  and  surface  commercial  vessels  are  transiting  in  the  vicinity. 

Command  responsibilities  are  as  follows: 

•  Commander  of  the  Task  Group  (CTG):  Embarked  in  Algonquin 

•  Anti-Air  Warfare  Commander  (AAWC):  Algonquin 

•  Anti-Submarine  Warfare  Commander  (ASWC):  CTG  embarked  in  Algonquin 

•  Anti-Surface  Warfare  Commander  (ASuWC):  CTG  embarked  in  Algonquin 
Sector  assignments  and  the  disposition  of  the  TG  are  shown  in  Figure  3.1. 

Without  our  knowing  the  full  ROE,  we  have  assumed  that  all  enemy  units  are  to  be  regarded  as 
hostile  and  to  be  engaged  and  destroyed.  The  enemy  is  known  to  have  submarines  in  the  area  and 
there  are  several  enemy  surface  groups  in  the  vicinity.  The  Task  Group  also  has  its  own  MPA 
assets  and  the  capability  to  pass  over-the-horizon-targeting  information  for  co-ordinated 
engagements. 

4.4.2.1  Terminology 

Throughout  the  report  to  avoid  confusion,  we  will  distinguish  between  the  Halifax  Anti  Submarine 
Warfare  Director  (ASWD)  and  The  TG  Anti  Submarine  Warfare  Commander  (ASWC),  which 
differs  from  the  terminology  that  has  been  used  in  previous  reports.  The  acronym  AS  will  refer  to 
the  TG  Anti  Surface  Warfare  Commander  (Also  ASuWC)  and  AX  as  the  TG  Anti  Submarine 
Warfare  Commander. 
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Figure  4.1:  sector  assignments  and  disposition  of  the  Task  Group 


4.4.3  Equipment 

Data  collection  took  place  in  the  briefing  room  of  the  ORTT  trainer.  The  briefing  room  is  used 
both  for  briefing  Ops  Room  teams  during  their  training  sessions  in  ORTT,  and  for  students  and 
staff  to  monitor  the  Ops  Room  performance  during  a  scenario  run.  Two  projection  screens  and  a 
47”  CRT  monitor  are  located  at  the  front  of  the  briefing  room  and,  along  with  two  speakers 
mounted  on  the  wall,  allow  real-time  audio  and  video  monitoring  of  a  scenario  run.  At  the  back  of 
the  briefing  room,  there  is  a  workstation  that  includes: 

•  A  Pentium  I  PC  running  ORTT  "Debrief'  software  that  was  capable  of  reproducing 
exactly  the  Command  and  Control  System  (CCS)  display  for  any  team  position. 

•  Four  16”  monitors  connected  to  the  PC:  three  monitors  could  display  different  CCS 
screens,  while  the  fourth  was  used  to  interact  with  the  Debrief  software  to  control  the  data 
playback. 

•  A  desktop  speaker  for  replay  of  audio  communication. 

•  Two  video  cassette  recorders  that  will  play,  in  real-time,  audio/video  recordings  from  a 
video  camera  and  microphone  in  cubicle  1  during  scenario  playback. 
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CCS  displays  could  be  selected  for  each  of  the  following  Ops  Room  Team  members:  CO,  ORO, 
SWC,  ASWD,  ARRO  (formerly  RT1),  ASPO  (formerly  RT2),  TS,  SAC,  EWS. 

Audio  communications 

The  generic  network  of  communication  links  for  the  Halifax  class  ships  in  the  context  of  this 
scenario  comprises  the  following  external  and  internal  networks: 

External: 

•  TG  Command:  CO,  ORO 

•  AAW Coord:  SWC,  ARRO,  EWS 

•  ASW  Coord:  ASWC,  ASPO,  SAC 

•  TG  Reporting:  TS 

Internal: 

•  C&C-.  CO,  ORO,  00 W,  TS 

•  AWW Coord-.  SWC,  ARRO,  EWS,  FCS 

•  ASW  Coord-.  ASWD,  ASPO,  SAC,  SCS 

These  are  shown  organised  by  Ops  Room  personnel  position  in  the  following  table. 


Position 

Internal  (Primary) 

Internal 

(Secondary) 

External 

(Primary) 

External 

(Secondary) 

CO 

C&C 

AWW/ASW 

TG  Command 

AAW/ASW 

ORO 

C&C 

AWW/ASW 

TG  Command 

AAW/ASW 

SWC 

AWW 

C&C 

AAW  Coord 

TG  Reporting 

ASWD 

ASW 

C&C 

ASW  Coord 

TS 

C&C 

TG  Reporting 

ARRO 

AWW 

AAW  Coord 

ASPO 

ASW 

C&C 

ASW  Coord 

TG  Reporting 

EWS 

AWW 

AAW  Coord 

SCS 

Sonar 

ASW 

ASW  Coord 

FCS 

AWW 

SAC 

ASW 

C&C 

ASW  Coord 

TG  Reporting 

OOW 

C&C 

AWW/ASW 

ASW/AAW 

TG  Command/TG 
Reporting 

Table  4.2:  Internal  and  external  communication  networks  organized 

by  Ops  Room  position 

For  the  CO  and  ORO,  whether  they  are  on  AWW  or  ASW  internal  or  AAW/ASW  external  is  threat 
dependent.  The  same  applies  to  the  OOW,  but  the  OOW  can  also  monitor  more  circuits  at  once 
because  he  can  put  several  SHINCOMS  on  a  speaker.  The  OOW  is  always  on  C&C,  with  usually 
two  external  circuits  monitored  on  speaker.  The  OOW  will  not  speak  on  these  external  circuits, 
only  listen.  The  reason  the  ASPO  may  be  on  C&C  and  TG  Reporting  as  secondary  circuits  is  that 
when  there  is  no  ASW  threat  he  will  help  with  surface  picture  compilation,  which  happens  on  TG 
Reporting.  The  configuration  identified  for  the  SAC  is  in  an  ASW  threat  environment;  if  the  helo 
or  MPA  he  is  controlling  is  on  a  surface  search  mission,  he  will  probably  be  in  the  secondary 


Page  28 


COMDAT:  MOP  and  ORTT 


Hum'&nsystems  Incorporated 


configuration.  The  FCS  is  not  on  an  external  net.  The  SCS  is  normally  on  two  internal  nets,  going 
onto  the  external  only  occasionally.  Other  than  these  cases,  everyone  usually  is  monitoring  one 
internal  and  one  external  circuit  simultaneously. 

The  ORTT  Debrief  facility  only  permits  the  playback  of  one  of  these  communication  networks  at 
any  time.  For  the  purposes  of  the  pilot  study,  wall-mounted  speakers  or  PC  speakers  were  used  to 
monitor  the  playback  of  the  audio  communications. 

The  F1S1®  T  &  E  team  comprised  the  project  director  (M.  L.  Matthews),  a  consultant  (L.  Bruyn)  a 
Navy  SME  (R.  Keeble)  and  was  assisted  by  the  ORTT  manager.  The  team  used  a  laptop  computer 
to  record  data  extracted  from  the  scenario  playback  and  also  collected  printouts  from  a  printer 
located  in  the  control  room  to  capture  screens  (from  any  Ops  Room  CCS)  at  various  points  during 
scenario  playback  to  assist  in  understanding  and  analysing  the  tactical  picture. 

Keystroke  data 

A  further  source  of  scenario  data  produced  by  the  Debrief  software  is  a  time-ordered  sequence  of 
keystrokes  captured  from  each  workstation.  This  record  may  be  displayed  on  one  of  the  Debrief 
displays  but  there  is  no  control  over  the  format  of  the  output.  Thus,  it  is  not  possible  to  display  just 
the  keystrokes  only  from  one  position,  or  selected  keystrokes,  or  for  a  specified  time  interval.  This 
data  record  is  not  only  considered  to  be  confidential  but  also  cannot  be  exported  to  other  file 
formats  to  aid  in  analysis.  Other  than  reviewing  this  data  record  for  its  potential  to  yield  MOPs,  no 
attempt  was  made  to  analyse  the  keystroke  record. 

4.4.4  Procedure 

We  were  able  to  observe  approximately  three  hours  of  training  with  an  intact  ship's  Ops  Room 
Team.  Flowever,  the  scenario  observed  was  somewhat  different  from  the  regular  training  scenario 
in  that  it  required  the  ship  to  pursue  a  possible  hostile  contact  without  the  usual  context  of  a  TG. 
This  contrasts  with  a  typical  training  scenario  in  which  the  ship  under  training  responds  to  a  series 
of  air,  surface  and  sub-surface  contacts  and  threats  in  a  typical  TG  format.  The  more  typical 
training  scenario  was  considered  to  be  more  representative  and  appropriate  for  the  analysis  of 
MOPs  and  extraction  of  data.  Thus,  the  scenario  under  observation  did  not  allow  us  to  note 
specific  scenario  events  that  would  be  the  focus  of  MOP  analysis.  Instead,  we  used  a  record  of  a 
more  typical  TG  scenario  record  from  the  previous  day,  which  meant  that  there  was  some 
additional  analysis  overhead  in  first  reviewing  the  record  for  areas  of  interest  for  analysis. 

The  scenario  segment  was  about  120  minutes  in  duration  with  approximately  50  minutes  anti-air 
warfare,  30  minutes  anti-submarine  warfare  and  30  minutes  anti-surface  warfare.  As  previously 
noted,  we  focused  on  the  performance  of  Team  1  since  they  were  being  evaluated.  In  order  to 
capture  the  data,  we  reviewed  approximately  75  minutes  of  scenario  playback,  first  at  a  higher  level 
in  order  to  get  a  general  perspective  of  the  events  within  the  scenario  segment.  This  allowed  us  to 
select  the  events  for  which  we  wanted  to  gather  more  detailed  information.  Within  this  75-minute 
scenario  segment,  we  then  decided  to  focus  on,  and  attempt  to  extract,  data  for  one  air,  one  surface 
and  one  subsurface  series  of  contacts.  The  data  extraction  procedure  is  described  in  the  following 
section. 

During  playback  we  were  able  to  select  and  observe  the  CCS  display  for  three  different  Ops  Room 
team  members  and  simultaneously  monitor  one  communications  network.  We  also  had  the  ability 
to  pause  the  replay  and  change  the  ORTT  team  position  selected  on  any  CCS  display  and  change 
the  communications  network  at  any  point  during  scenario  playback. 
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The  CCS  displays  selected  depended  on  the  specific  scenario  event  and  the  key  team  members  for 
that  event,  as  follows: 

•  Air  contacts:  ORO,  SWC  and  ARRO  displays  (supplemented  by  the  CANEWS  operators 
or  SAC,  when  necessary) 

•  Surface  contacts:  ORO,  SWC  and  TrackSup 

•  Sub-surface:  ORO,  ASWD  and  ASPO 

Similarly,  the  communications  network  selected  for  monitoring  was  dependent  on  the  specific 
scenario  event.  For  example,  if  the  scenario  event  involved  significant  communication  between 
ships  in  the  TG,  the  TG  Command  network  would  be  monitored.  On  the  other  hand,  if  the  scenario 
event  involved  communications  about  a  missile  fire  by  ownship,  the  internal  AWW  Coord  network 
would  be  monitored. 

4.4.4.1  Data  Extraction 

For  the  purpose  of  the  pilot  study,  it  was  decided  that  relevant  data  would  be  extracted  from  the 
scenario  record  for  scenario  segments  that  focussed  on  air,  surface  and  subsurface  picture  building. 
Scenario  playback,  and  therefore  data  extraction,  for  each  event  started  just  prior  to  initial 
registration  of  the  air,  surface  and  subsurface  contact  on  the  CCS.  Scenario  playback  and  data 
extraction  for  each  scenario  event  ended  at  the  point  of  resolution  or  at  the  end  of  threat 
engagement  for  the  air,  surface  and  subsurface  contact.  In  total,  it  took  approximately  12  hours  of 
initial  data  extraction  by  3  persons  to  capture  75  minutes  of  the  scenario. 

As  previously  noted,  the  specific  CCS  displays  and  communications  network  being  monitored  were 
dependent  on  the  specific  scenario  events  and  hence  varied  throughout  the  data  extraction  process. 

In  order  to  extract  sufficient  data  for  each  contact  from  detection  to  resolution,  it  was  necessary  to 
repeatedly  pause,  analyse  and  note,  then  rewind  and  playback  each  scenario  event  or  contact.  This 
iterative  analysis  required  observing  different  CCS  displays  or  monitoring  a  different 
communications  network,  in  order  to  extract  all  of  the  pertinent  information  for  an  event.  During 
this  process  co-ordination  and  registration  between  the  different  CCS  displays  was  maintained. 

The  data  extracted  from  video  (CCS  display)  and  audio  playback  of  scenario  events  was  logged 
directly  into  a  spreadsheet  by  the  T  &  E  team.  This  included: 

•  Network  communications  including  the  specific  network  over  which  the 
communication  occurred  (e.g.  C  &  C,  TG  Command),  the  identity  of  the  person  sending 
the  communication,  the  identity  of  the  person  receiving  the  communication  and  the 
content  of  the  communication. 

•  Local  picture  information,  such  as  range  scale,  at  critical  points  in  the  scenario 
playback  (printouts  of  the  CCS  display  provided  this  information  as  well). 

•  Hooked  tracks  at  critical  points  in  the  scenario  playback. 

•  Time  of  occurrence  for  each  network  communication,  local  picture  change,  keystroke, 
or  significant  event  logged. 

•  Additional  comments  noted  by  the  experimenter  for  future  reference.  For  example,  an 
Ops  Room  Team  member  might  have  been  on  a  different  network  channel  and  therefore 
missed  an  attempted  communication  from  another  Ops  Room  Team  member.  This 
represents  important  information  that  would  not  be  captured  by  the  above  data  extraction 
methods. 
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The  printouts  of  a  specific  operator’s  CCS  display  at  critical  points  in  the  scenario  playback 
provided  more  detailed  local  picture  information  and  could  be  used  to  validate  local  picture 
information  logged  in  the  spreadsheet  by  the  T  &  E  team. 

Although  our  objective  was  to  focus  on  performance,  specifically  actions  and  communications,  of 
the  ORO,  it  was  evident  that  actions  and  communications  of  other  ownship  Ops  Room  team 
members  and  other  TG  members  either  directly  or  indirectly  affect  the  ORO’s  performance. 

Further,  for  the  purpose  of  evaluating  the  potential  impact  of  MSDF  technology,  it  was  necessary 
to  also  analyse  tasks  of  other  teams  members  in  the  picture  building  and  contact/threat 
identification  process  as  well  as  associated  communications.  Therefore  the  decision  was  made  to 
extract  data  for  all  relevant  ownship  Ops  Room  Team  members  and  TG  members  for  each  scenario 
event.  This  then  allows  us  to  consider  the  context  surrounding  the  ORO’s  performance.  For 
example,  the  time  it  takes  the  ORO  to  interrogate  a  contact  may  be  affected  by  the  fact  that  the 
SWC  tried  to  provide  information  to  the  ORO  on  a  network  channel  that  the  ORO  is  not  listening 
to.  This  could  only  be  determined  by  logging  the  network  communications  of  the  SWC  rather  than 
those  of  the  ORO  alone,  and  analysing  overall  information  exchange  patterns  among  team 
members  and  their  associated  timeliness  and  errors. 

4.5  Results 

The  data  record  collected  in  the  manner  described  above  comprised  an  Excel  spreadsheet  file.  This 
was  reviewed  by  the  HSI®  team  and  EMC  ORTT  support  personnel  and  deemed  not  to  contain  any 
confidential  data. 

For  the  purpose  of  this  report,  three  edited  and  selected  segments  of  the  data  record  are  included  in 
Annex  C,  each  corresponding  to  a  separate  series  of  events  in  air,  surface  and  sub-surface.  The  full 
record  has  been  provided  to  the  Scientific  Authority  separately  from  the  report  in  electronic  format 
as  an  Excel  spreadsheet. 

The  results  will  be  presented  as  follows.  First  we  will  provide  an  overview  and  detailed  narrative 
of  each  event  series  (air,  sub-surface,  surface)  with  reference  to  the  data  record.  These  are 
described  in  the  chronological  order  in  which  they  occurred  in  the  scenario.  Second,  we  will 
provide  concrete  examples  of  MOP  data  extracted  from  each  event  series.  Third,  we  will  comment 
upon  other  potential  MOPs  that  might  be  extracted.  Finally,  we  will  review  the  preliminary  MOP 
list  outlined  in  previous  reports  and  provide  a  summary  of  the  potential  and  suitability  of  the  ORTT 
to  facilitate  data  collection  and  extraction  relevant  to  each  MOP. 

It  will  become  apparent  in  subsequent  sections  that  the  full  communication  record  was  not  analysed 
in  as  much  detail  as  the  record  of  CCS  events  captured  in  the  data  record.  For  matters  of 
expediency  and  limited  time  available  we  concentrated  on  the  CCS  record.  This  should  not  be 
taken  to  mean  that  we  believe  that  the  communication  record  is  neither  worth  analysing  nor 
amenable  to  analysis.  We  believe  that  the  total  time  spent  in  communications,  communication 
errors  and  communication  latencies  all  represent  important  MOPs  that  may  reflect  the  impact  of 
MSDF  technology.  Indeed,  in  some  instances  communication  measures  may  provide  the 
appropriate  level  of  diagnosis  and  analysis  concerning  how  the  technology  impacts  on  the  detect- 
to-resolve  cycle. 
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4.5.1  Air  event  series:  description  and  overview 

Although  the  following  narrative  concentrates  largely  on  air  warfare  events,  other  relevant 
information  from  other  warfare  domains  will  be  interleaved  as  they  relate  to  the  ORO's  picture 
building,  tactical  planning  and  situation  awareness.  In  order  to  understand  the  relationship  among 
the  various  main  events  and  actions  a  GANTT  style  chart  (Figure  2)  has  been  prepared.  This 
shows  the  time  of  onset  and  duration  of  each  of  the  new,  main,  contact  events,  significant  responses 
from  the  Ops  Room  team,  the  ORO's  picture  (as  exemplified  by  his  selected  range  scale)  and  the 
ORO's  communications.  For  clarity,  existing  contacts  on  the  CCS  plot  at  the  start  of  the  scenario 
series  have  not  been  included. 
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Figure  4.2:  Gantt  chart  showing  main  events  and  actions  for  air  contacts 
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Figure  4.3:  ORO  CCS  Display  at  Scenario  Start 

The  segment  for  analysis  starts  soon  after  the  scenario  has  commenced.  At  the  start  of  the  segment, 
the  ORO  CCS  display  is  centred  upon  the  area  of  operations  at  a  range  of  256  nm  (see  Figure  3). 
Fligh  level  air  lanes  (solid,  straight  yellow  lines)  and  Areas  of  Probability  AOPs  (yellow  circles) 
associated  with  known  submarines  (one  of  which  is  believed  to  be  an  OSCAR  type,  based  on 
intelligence)  and  SAGs  are  shown  in  yellow.  The  ORO  has  a  surface  consort  hooked  (box  around 
contact)  and  there  are  at  least  one  commercial  (green  track  number  1632)  aircraft  in  the  air  lanes. 
This  suggests  that  at  present  the  ORO  is  building/maintaining  the  global  picture.  The  Task  Group 
has  two  sub-surface  AOPs  close  by,  based  on  time  late  locating  information  provided  by 
intelligence.  Then: 

•  At  9: 18.44,  a  new,  unassessed  air  track  (2400)  appears  NW  of  the  TG,  which  the  ORO 
immediately  hooks. 

•  The  SWC  who  is  ranged  at  128nm,  orders  resolve  on  track  2400. 

•  Within  10  seconds,  the  ARRO  reports  on  the  AWW  net  no  IFF  and  4  seconds  later 
CANEWS  reports  no  EW. 
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•  Two  seconds  later  the  track  is  made  suspect  the  SWC  orders  that  "warning  #1 "  be 
issued8. 

•  The  ship  then  receives  a  brief  SITREP  from  the  AAWC  in  ALG. 

•  Five  seconds  later,  the  ORO  hooks  track  2400  and  ranges  down  to  128  nm,  which  would 
indicate  shifting  to  the  local  picture  and  paying  attention  to  this  potential  threat. 

•  At  the  same  time  the  ORO  announces  that  he  is  going  external  from  the  C&C  net. 

•  While  he  is  external,  the  SWC  provides  a  SITREP:  a  general  assessment  as  well  as 
comments  on  track  2400. 

•  There  is  no  communication  to  indicate  that  the  ORO  is  "back  from  external",  so  we  might 
assume  that  he  misses  this  SITREP  from  the  SWC. 

•  At  9:21.40,  three  minutes  later,  the  ORO  asks  the  SWC  if  there  are  any  surface  contacts 
that  2400  may  be  passing  targeting  information  to. 

This  indicates  that  the  ORO  is  thinking  about  the  tactical  implications  of  2400  and  integrating  the 
air  and  surface  pictures.  Of  interest  is  the  fact  that  the  SWC  misunderstands  this  question  and 
agrees  with  the  ORO  that  2400  is  an  MPA.  This  suggests  that  the  SWC  is  thinking  more  locally 
about  track  2400  and  not  the  surface  picture. 

•  Five  seconds  later,  the  ORO's  attention  is  shifted  to  the  sub-surface  picture  by  a  routine 
SITREP  from  the  ASWD  and  the  ORO  prompts  with  a  request  for  more  information 
about  AOPs. 

Something  in  the  SITREP  causes  the  ORO  to  order  a  second  torpedo  loaded  onto  one  of  the  helos. 

In  fact,  a  few  seconds  later  a  sub-surface  track  (2403)  appears  on  LINK  from  REG.  Thus,  in  the 
space  of  20  seconds,  the  ORO  has  shifted  from  the  air/surface  picture  to  the  sub-surface  picture  and 
shows  evidence  of  planning  ahead. 

•  Further  evidence  for  this  is  found  at  9:22:19  when  the  ORO  hooks  the  AOP  associated 
with  a  submarine. 

•  Five  seconds  later,  the  ORO's  attention  is  drawn  back  to  the  air  picture,  when  the  SWC 
announces  that  "warning  3"  is  to  be  issued  to  track  2400. 

Remember  that  only  "warning  1"  had  been  previously  issued.  We  were  not  able  to  find 
anything  in  the  communication  record  between  the  SWC  and  ORO  concerning  "warning  2",  so 
there  seems  to  be  some  communication  failure  here  which  would  critically  impact  upon  the 
ORO's  appreciation  of  the  air  picture  and  the  urgency  of  the  potential  threat  from  2400. 

•  At  9:24.30,  further  evidence  of  this  confusion  is  shown  when  the  SWC  informs  the  ORO 
that  they  have  been  ordered  to  issue  "warning  2"  to  2400! 9 

•  At  9:23:04  a  new  air  track  (2404)  by  link  from  REG  and  a  few  seconds  later  HAL 
generates  a  local  track  (8011)  -  within  one  minute  these  two  tracks  are  correlated  by  the 
ARRO. 

This  has  happened  without  any  indication  of  ORO  interest  (which  would  be  expected  given  that  the 
track  is  commercial  air),  who  is  now  off  the  C&C  net  and  talking  with  the  ASWC.  The  ORO 


8  The  direction  for  this  would  normally  have  come  from  the  AAWC,  or  it  may  have  been  directed  in  the  OPTASK  AAW  to 
automatically  issue  warning  1 .  Whichever  the  case,  this  issue  was  not  considered  sufficiently  important  for  present  purposes  for  us  to 
spend  time  tracking  down  this  information. 

9  The  order  to  issue  this  warning  would  come  from  the  AAWC  in  ALG. 
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appears  to  be  in  the  process  of  sub-surface  picture  building  and  talking  to  the  CO  at  this  time  about 
the  position  of  HAL  in  relation  to  the  sub  bullpen. 

•  At  9:24:24  a  new  air  track  (8013-missile-  30  nm  NE)  appears  which  is  hooked  within  3 
seconds  by  the  ORO  who  then  receives  the  message  from  the  SWC  about  warning  2  to 
track  2400.  The  track  is  immediately  identified  as  a  missile  because  of  speed. 

The  ORO  directs  the  SWC's  attention  to  the  incoming  missile  who  immediately  initiates  a  resolve 
on  this  track.  The  ORO  then  contacts  the  OOW  concerning  the  location  of  the  incoming  missiles 
and  a  few  seconds  later  the  ship  is  brought  to  Action  Stations. 

•  Ten  seconds  later  the  SWC  reports  that  the  tracks  are  assessed  as  missiles,  based  upon 
speed  and  ESM,  and  three  seconds  later  the  ORO  agrees  with  this  assessment  and  the 
ORO's  array  selection  shows  that  he  is  ready  to  fire  against  the  incoming  missile. 

•  Forty  seconds  later,  the  SWC  hits  the  fire  button  to  engage  the  missile,  but 
unknowingly  is  on  the  wrong  page  of  his  input  array.10  Although  there  is  a  call  "missile 
away"  on  one  of  the  nets,  unbeknown  to  the  SWC  it  is  the  ORO  who  has  fired  the 
missile. 

Apparently,  the  ORO  was  able  to  monitor  the  SWC's  firing,  recognise  the  error  and  step  in  to 
rectify  the  situation  within  the  space  of  15  seconds.  The  elapsed  time  from  the  SWC  initiating  the 
firing  and  the  missiles  getting  away  is  33  seconds.  Ten  seconds  after  the  first  missile  launching, 
the  SWC  successfully  gets  away  the  second  missile. 

Of  interest,  just  before  the  ORO  is  about  to  initiate  the  launch  of  the  first  missile,  the  ASWD  asks 
to  give  a  SITREP  -  which  the  ORO  unsurprisingly  ignores.  Although  the  ASWD  will  have  heard 
the  action  alarm  and  pipe,  he  appears  to  be  unaware  that  this  is  an  inopportune  moment  to  be 
delivering  a  SITREP. 

•  At  9:27:00,  following  the  engagement,  the  ORO  directs  the  OOW  to  resume  patrolling 
the  sector  and  ranges  out  to  128  nm  to  resume  building  the  larger  picture. 

•  During  this  time  of  the  missile  attack  the  assessment  of  Track  2400  has  been 
continuing  and  at  9:27.08  EWS  reports  EW  racket  which  correlates  with  Track  2400 
and  identifies  contact  as  "Mainstay. 

•  Track  2404  is  again  reported  by  LINK  from  REG  and  for  a  while  there  is  some  "dual 
tracking"11  as  HAL  is  holding  a  local  track  on  it. 

•  At  9:29:30,  after  about  30  seconds  of  dual  tracking,  the  ARRO  correlates  the  two 
tracks  and  the  contact  is  made  Assumed  Friend. 

During  this  period,  there  is  no  evidence  that  the  ORO  has  been  following  the  progress  of  2400  and 
2404.  Instead,  he  appears  to  be  still  concerned  about  building  the  ASW  picture. 

•  At  9:27:50  the  ORO  has  a  discussion  with  the  ASWD  to  refine  the  AOP  for  the 
OSCAR  sub,  based  upon  the  assessment  that  it  was  responsible  for  the  SSI 9  missile 
attack.  Within  10  seconds  he  then  switches  back  to  the  air  picture  and  hooks  track 
2400. 

•  At  9:28:50,  one  minute  later,  a  new  air  track  (2412)  is  generated  by  REG. 


10  We  were  able  to  determine  this  by  comparing  the  array  selection  of  the  ORO  and  SWC  at  the  point  of  firing  the  missiles. 

1 1  By  dual  tracking  in  this  instance  we  mean  that  there  two  tracks  have  been  generated  for  probably  the  same  target  from  different  sensor 
systems  or  platforms 
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•  Fifteen  seconds  later  2400  is  made  hostile  by  REG  and  the  ORO  orders  on  the  C&C 
net  that  2400  be  made  hostile  and  is  assessed  as  "Mainstay"  (a  known,  potentially 
hostile,  air  platform)12. 

•  At  9:29.20  a  "zippo"  call  is  made  by  the  EWS  on  a  specific  EW  racket  and  a  recon 
track  is  put  in  on  the  appropriate  bearing.  The  zippo  call  reflects  a  positive 
identification  of  an  aircraft  radar  that  is  associated  with  an  air  to  surface  missile. 

•  Sixteen  seconds  later,  HAL's  SGI 50  generates  local  air  tracks  (8032  and  8033)  which 
are  released  to  LINK  as  hostiles  2222  and  2223.  Of  interest,  during  this  important 
change  in  threat  status,  the  ORO  has  had  no  direct  role  in  the  process  but  has  been 
monitoring  the  associated  air  picture. 

•  Within  30  seconds,  the  EWS  recommends  that  the  tracks  are  aircraft  not  missiles  based 
on  course  and  speed  alterations  -  the  SWC  agrees. 

•  Eighteen  seconds  later,  evidence  is  seen  on  the  SG  150  radar  of  chaff  released  in  the 
vicinity  of  these  tracks;  five  seconds  later  the  ORO  receives  an  order  from  the  CO  to 
change  the  missile  firing  parameters  from  a  salvo  engagement  to  a  single  shot 
engagement. 

•  After  the  firing  of  the  missiles  the  ORO  ranges  down  to  1 6nm  and  then  8nm  to  follow 
the  engagement. 

Of  interest,  the  ASWD  also  follows  this  engagement  down  to  the  same  range,  rather  than 
concentrating  on  the  AOP  for  which  he  has  primary  responsibility.  The  total  time  from  the  initial 
contact  of  the  two  unknown  contacts  to  the  decision  as  to  how  many  Sea  Sparrows  to  use  against 
the  contacts  was  one  minute. 

•  Towards  the  end  of  this  engagement,  two  new  air  tracks  (2224/5)  have  appeared  and 
been  interrogated  on  IFF  by  the  ASPO  (the  ARRO  was  presumably  following  the 
ongoing  engagement)  who  makes  them  friendly  based  on  Mode  4. 

•  Thirty  seconds  later  the  ARRO  ranges  out  and  picks  up  these  tracks.  There  appears  to 
be  no  discussion  on  the  HAL  nets  among  the  team  about  these  contacts.  At  9:35.20  the 
AAWC  advises  HAL  that  he  is  sending  the  CAP  (2224/5)  to  attack  2400. 

•  At  9:36  the  SWC  provided  a  SITREP  to  the  ORO,  recommending  return  of  all 
equipment  to  baseline  and  updates  status  of  missile  inventory. 

4. 5. 1. 1  Analysis  of  ORO 's  air  contact  activities 

This  sequence  of  events  underlines  the  findings  derived  from  the  CTA  of  the  ORO  (Matthews, 
Webb  &  Bryant,  1999).  Namely,  that  the  ORO  is  constantly  switching  between  many  pictures  and 
levels  of  the  tactical  environment  as  he  simultaneously  monitors  and  guides  all  Ops  Room 
processes,  seeks  and  integrates  data  to  build  an  integrated  view  of  the  combined  tactical  picture, 
and  plans  ahead. 

It  would  appear  that  the  ORO's  highest  priority  concerns  sub-surface  threats,  as  there  appears  to  be 
ongoing  uncertainty  concerning  their  specific  location,  proximity  and  hence  danger  to  the  TG. 

This  priority  is  evidenced  by  his  several  conversations  with  other  team  members  concerning  the 
sub-surface  picture  and  his  plans  for  launching  a  helo  to  improve  this  picture  and  possibly  deal 


12 

There  is  some  ambiguity  in  the  interpretation  of  this  event  which  may  not  be  resolved  without  further  delving  into  the  communication 
record.  Earlier  HAL’s  EWS  had  reported  a  Mainstay  racket  that  correlated  with  2400.  HAL  should  have  made  2400  Hostile  based  on 
that,  but  apparently  didn’t.  If  REG  is  the  one  that  made  2400  hostile,  then  HAL’s  SWC  would  not  order  on  C&C  to  make  2400  hostile  as 
it  already  would  be.  Perhaps  instead  he  reported  on  C&C  that  2400  was  hostile. 
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with  any  threats.  Note  that  the  ORO  was  previously  ordered  by  CTG  to  re-arm/launch  his  helo. 
Thus,  the  ORO  is  reflecting  the  high  priority  that  the  ASWC  has  regarding  the  ASW  threat,  and  is 
weighing  this  concern  against  the  immediate  air  threat.  The  ORO  also  shows  clear  evidence  of 
integrating  the  tactical  pictures  across  warfare  domains  in  his  concern  over  the  role  that  contact 
2400  (a  potentially  hostile  MPA)  may  be  playing  in  passing  targeting  information  to  enemy 
submarines  or  SAGs. 

The  ORO's  overall  concern  for  the  sub-surface  picture  is  punctuated  by  shifts  in  focus  to  monitor 
short-term,  urgent  air  threats.  In  these  cases,  his  role  tends  to  be  that  of  close  supervision  and 
management  of  processes  rather  than  being  directly  involved  in  the  detect-to-resolve-to-engage 
activities.  His  awareness  of  an  error  in  these  processes  is  evidenced  by  his  immediate  taking  over 
of  the  firing  of  the  first  missile  against  the  incoming  SS-19,  when  the  SWC  failed  to  initiate  the 
firing. 

4.5.1. 2  MOPs 

For  clarity,  we  divide  the  MOPs  into  two  categories: 

•  MOPs  centred  on  the  ORO  and  derived  from  the  CTA  and  subsequent  analyses,  and 

•  MOPs  oriented  towards  COMDAT  MSDF  and  the  impact  of  that  technology  on  the 
D2R  process. 

Because  of  the  limited  time  available  for  analysis  of  all  of  the  MOP  data  that  was  potentially 
available,  illustrative  examples  of  as  many  MOPs  as  possible  that  could  be  readily  extracted  are 
provided. 

For  consistency,  task  and  MOP  numbers  refer  to  MOP  tables  in  prior  reports. 

We  define  key  MOP  terms  as  follows: 

Accuracy:  usually  percent  correct  -  number  of  correct  items/number  of  opportunities  (example:  ten 
hostile  a/c  appear  during  scenario-  team  correctly  identifies  8  -  hence  accuracy  is  80%). 

Response  time:  usually  the  time  from  the  onset  of  an  event  as  defined  by  a  recognised  trigger  point 
for  the  team.  (For  example,  this  could  be  the  appearance  of  a  CCS  contact  or  a  message  on  the  net) 
until  a  recognisable  action  is  taken  that  terminates  the  event  (a  contact  is  made  suspect,  friend  or 
hostile).  (Example:  time  from  detection  of  new  underwater  sonar  contact  to  the  call  "Poss  sub  1”  is 
25  seconds). 

Communication:  typical  measures  include  the  total  number  and/or  time  spent  in  comms  around  a 
specific  event.  For  example,  eight  queries  among  AAW  to  identify  a  contact;  total  time  in  queries 
40  seconds.  Other  measures  include  the  rate  of  occurrence  of  errors  (number  of  error  messages 
expressed  as  a  percentage  of  the  total  number  of  messages)  and  the  rate  of  requests  for  repeated 
messages  (number  of  requests  expressed  as  a  percentage  of  the  total  number  of  messages). 


ORO  centred  MOPs 


DETAILED  MEASURES  EXAMPLE  from  SCENARIO 

COMMENTS 

T&E  task  1.  Check  mission  picture 

We  did  not  request  or  have  access  to 
paper  based  messages;  this  task  was 
given  zero  priority  because  of  the 
limited  scope  of  the  pilot. 
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DETAILED  MEASURES 

EXAMPLE  from  SCENARIO 

COMMENTS 

la  Accuracy  in  detecting  relevant  info 
within  incoming  message  stream 

1 .  AAW  (ALG)  provides  SITREP:  60 
seconds  later  ORO  ranges  down  to  AOI 
of  SITREP. 

2.  AAW -AAWC  advises  of  CAP.  ORO 
immediately  ranges  down  and  selects  this 
area  of  focus. 

Time  ref:  9:19:14 

Time  ref:9:35:20 

Accuracy  is  assessed  in  terms  of  the 
ORO  correctly  taking  an  appropriate 
action  based  upon  the  message 
content, 

Id  Time  to  detect  high  salience  message 
(i.e.  of  immediate  relevance  to  context) 

Less  than  1  sec  (example  2  above) 

If  Accuracy  in  ignoring  lower  priority 
messages 

ORO  ignores  untimely  SITREP  from  ASWD 
when  about  to  fire  missile. 

Time  ref  9:25:52 

TASK  2.  Relate  new  info  Ops  Room  picture 

2a  Accuracy  in  directing  (communicates) 
info/direction  resulting  from  message 

Error  example:  CO  reminds  ORO  about  equipt 
malfunction  on  a  hostile  unit:  ORO  does  not 
communicate  implications  to  team. 

Time  ref:  9:36:50 

2c  Accuracy  in  comprehension  of  impact 
on  pre-plans/response  options/tactical 
situation 

No  specific  examples  observed 

Table  4.3:  Examples  of  ORO  centred  MOPs  collected  during  Pilot  Trial 

(continued  on  next  page) 
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DETAILED  MEASURES 

EXAMPLE  from  SCENARIO 

COMMENTS 

TASK  3  Relate  (new  info)  to  RMP 

Includes  aspects  of  build  awareness  across  domains  (RAP,  MSP,  MsubP). 

Generally  requires  probe  method  not 
available  in  ORTT 

3d  Taking  appropriate  action  as  a  result  of 
updated  RMP 

ORO  has  been  monitoring  an  unknown  contact 
(2207).  ORO  immediately  calls  to  SWC 
attention,  trk  gets  ID  as  incoming  missile  ORO 
advises  OOW  where  to  monitor  and  advises 

CO  for  action  stations. 

Time  ref:  9:24.24-  9:24.50. 

Entire  sequence  takes  37  seconds. 

3f  Accuracy  in  assessing  the  integrated 
tactical  picture 

ORO  considers  role  of  2400  in  possibly 
relaying  targeting  information  to  sub-surface 
threat. 

The  analysis  was  not  followed 
through  to  determine  if  this  was  an 
accurate  assessment.  Although  is 
clear  that  the  ORO  has  considered 
this  as  a  possibility. 

TASK  7.1  Detect  changes  in  tactical  situation 

7.1c  Time  to  detect  new  threat 

8013-  hooked  in  3  secs 

9:24.24 

7.1  d  Time  to  detect  change  in  threat  status 

2400  from  unknown  to  hostile:  38  secs 

9:28:30.  This  is  the  time  between  the 
EWS  making  the  correlation  and  REG 
making  it  hostile.  In  this  particular 
context  the  MOP  reflects  TG  AAW 
team  performance  rather  than  that  of 
the  ORO  on  HAL. 

TASK  7.2  Determine  threat  priorities  across  domains 

7.2a  Accuracy  in  ranking  threat 

priorities  across  domains 

Switches  from  assessing  sub-surface  threat 
(not  imminent)  to  more  immediate  air  threat. 

9:24.24 

9:29:15 

During  time  of  potential  threats  on 
both  surface  and  sub-surface,  the 

ORO  and  CO  concentrated  on 
surface  action  and  left  ASWC  to 
handle  sub-surface. 

SUB  TASK  7.3  Assess  threats-within  a  specific  warfare  domain.  Identify  track  threat 
priority 

7.3a  Accuracy  in  identifying  threat 

priorities 

Several  examples  of  where  ORO  changed 
focus  to  deal  with  more  immediate  threat.  No 
errors  detected. 

A  more  complex  example  was  when 
two  aircraft  (222/3)  had  initially  been 
assessed  as  missiles;  once  they  were 

ID  as  aircraft,  (the  ORO  observed 
that  they  were  jinking)  the  CO 
changed  from  salvo  to  single  shot 
because  they  were  easier  targets. 

TASK  10  Manage  ship  surveillance 

10a  Accuracy  recognising  problems  in 
detect  to  resolve  process 

Example  1 :  One  error  in  ID  went  undetected  by 
the  SWC  and  ORO  for  several  minutes. 

Example  2:  Error  in  monitoring  warning 
sequence  to  2400 

Notes  taken  during  scenario. 

9:19.09  &  9:22.45 

10b  Time  to  recognise  problems  in  detect- 
to-resolve  process 

Trk  2207: 8  secs  to  recognise  that  resolve  has 
not  started  on  this  and  notifies  SWC 

9:24.27 

Table  4.3:  Examples  of  ORO  centred  MOPs  collected  during  Pilot  Trial 
(continued  from  previous  page) 

It  should  be  remembered  that  the  above  MOP  areas  were  selected  on  the  basis  of  their  relevance  to 
ORO  picture  building  and  the  potential  impact  of  MSDF  technology.  MOPs  related  to  ORO 
management  of  processes,  teams  and  responses  to  threats  were  assigned  a  lower  priority  for  the 
present  trial.  However,  it  should  be  noted  that  we  were  able  to  observe  events  and  ORO  actions 
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that  would  support  the  collection  of  MOPs  in  these  areas.  For  example,  during  a  missile 
engagement  (reference  9:25.40)  the  ORO  accurately  monitored  a  failed  missile  firing  by  the  SWC 
and  took  over  the  missile  launch. 


MOPs  for  contact  detection  &  identification  (air  warfare  examples) 


DETAILED  MEASURES  EXAMPLE  from  SCENARIO 

COMMENTS 

TASK  4.1  Identify  friendly  aircraft 

Several  occurrences  in  observed 
scenario.  N=2  examined  in  detail 

4.1a  Accuracy  in  identifying  friendly  aircraft 

One  target  ID  correctly  immediately,  one  ID  after 
several  minutes  on  the  radar  plot.  Final  accuracy 
rate  =  100%.,  but  one  error  in  ID  went  undetected 
by  the  SWC  and  ORO  for  several  minutes. 

ASPO  made  trks  2224/5  friendly. 

These  tracks  not  noticed  by  ARRO 
until  31  sec  later. 

4.1b  Mean  time  spent  in  identifying  friendly 
aircraft 

Time  to  ID  single  friendly  a/c=44  sec  (Trk2412) 

9:28.50 

May  not  be  a  representative  measure 
as  rate  of  background  air  traffic  was 
low  for  this  scenario 

TASK  4.2  Identify  hostile/suspect  aircraft 

4.2a  Accuracy  in  identifying  hostile  / 
suspect  aircraft 

Accuracy  100%.  All  hostile  AC  were  eventually 
identified;  in  some  cases  they  remained  suspect 
for  a  time  before  being  correctly  ID.  No  hostile  a/c 
was  ID  as  friendly. 

Trks  2222/3:  initially  suspected  as  missiles 
assessed  quickly  as  a/c 

Trks  2400,  2222/3.  This  delay  in 
making  Hostile  is  not  to  suggest  a 
failure  in  the  team.  The  appropriate 
criteria  for  Hostile  were  not  achieved 
until  the  ESM  intercept  on  the 

Mainstay 

4.2b  Mean  time  to  identify  hostile/suspect 
aircraft 

Example  1:  Trk  2400 

Time  to  make  suspect=29  sec 

Time  to  make  hostile  10:08  (by  REG) 

Time  to  ID  as  hostile:  9  minutes-14  secs  based  on 
EWS  correlation  TRK221 7 

Example  2:  Trks  2222/3 

SWC  changed  prelim  ID  from  exocet  to  a/c  in  22 
sec  based  on  flight  pattern. 

9:18.40 

Initially  correctly  ID  as  suspect- 
subsequently  correctly  ID  as  hostile 
based  upon  new,  more  detailed 
information. 

9:32.40 

4.2c  Total  number  of  queries  or  total  time 
spent  in  querying  team  for  additional 
info 

Example  1  Trk  2400:  Eight  communications 
among  AAW  team 

Example  2:  One  error  of  miscommunication 
between  ORO  and  SWC  (error  was  semantic 
rather  than  a  missed  comm) 

Comms  involving  the  SAC  -  not 
counted 

TASK  4.3  Identify  neutral  aircraft 

Did  not  capture  this  event  in 
scenario 

TASK  4.4  Identify  NU  tracks  (non-updated) 

Not  normally  Included  in  scenario 
in  ORTT 

Table  4.4:  Examples  of  Detect  to  Resolve  MOPs  (air  contacts)  collected 
during  Pilot  Trial  (continued  on  next  page) 
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DETAILED  MEASURES  EXAMPLE  from  SCENARIO 

COMMENTS 

TASK  4.5  Identify  tracks  reported  by  ownship  or,  conversely,  by 

other  participating  units  (Pus) 

Not  relevant  in  this  scenario 
because  of  TG 
taskings/responsibilities. 

4.5a  Accuracy  identifying  air  tracks  being 
reported  by  ownship 

Not  a  relevant  measure 

4.5b  Total  time  spent  locating  air  tracks 
being  reported  by  ownship 

Not  a  relevant  measure 

4.5c  Total  number  of  queries  or  total  time 
spent  in  querying  team  for  additional 
info 

Not  a  relevant  measure 

TASK  4.7  React  to  threat  track  symbology  LINKed  by  consort  not  on  video 

4.7a  Time  to  recognise  symbology  not  on 
video 

N/A  in  ORTT 

4.7b  Time  to  order  remedial  action 

N/A  in  ORTT 

TASK  4.8  React  to  LINK  not  gridlocked 

4.8a  Time  to  recognise  LINK  not 
gridlocked 

N/A  in  ORTT 

4.8b  Time  to  order  remedial  action 

N/A  in  ORTT 

TASK  4.9  Resolve  ambiguous/dual  tracks 

Accuracy  in  resolving  dual  tracks 

Example  1 :  Linked  trk  2404/local  trk801 1 . 

Example  2:  Linked  trk  2404/localtrk8027 

Time  ref:  9:23.04 

Time  ref:  9:27.56 

Time  required  to  resolve  dual  tracks 

Example  1:  Linked  trk  2404/local  trk801 1 .  51  secs 
-  manually  by  ARRO 

Example  2:  Trks  2404/8027  -  81  secs  manually  by 
ARRO 

Table  4.4:  Examples  of  Detect  to  Resolve  MOPs  (air  contacts)  collected 
during  Pilot  Trial  (continued  from  previous  page) 

4.5.2  Sub-surface  event  series:  description  and  overview 

The  relationship  among  the  various  main  events  and  actions  for  subsurface  contacts  is  show  in  a 
second  GANTT  style  chart  (Figure  4).  There  are  two  submarine  threats  in  this  scenario,  an 
OSCAR  class  anti-ship  missile  firing  submarine  and  an  AKULA  class  attack  submarine.  The  Task 
Group  has  two  AOPs  close  by,  based  on  time  late  locating  information  provided  by  intelligence. 
The  earlier  SSN  19  missile  attack  has  provided  evidence  that  the  OSCAR  submarine  at  least  is  in 
close  proximity. 

•  At  10:00  the  ASWC  on  Flalifax  reported  a  sonar  contact,  Track  2275,  at  3  NM  from 
Flalifax.  It  was  almost  immediately  classified  as  "poss  sub  low  2"  and  Flalifax 
responded  by  conducting  an  urgent  toipedo  attack  in  accordance  with  the  preplan. 

•  Approximately  30  seconds  later,  Flalifax  ASWD  changed  Track  2275  classification 
from  "poss  sub  low  2"  to  "poss  sub  low  1".  It  is  possible  that  it  may  have  been 
incorrectly  entered  into  the  system  previously. 

•  At  10:02  HAL  then  reassessed  Track  2275  as  poss  sub  low  2.  HAL  then  initiated  a 
cordon  search  and  attack  plan  (comprising  HAL,  REG  and  associated  helo  assets)  in 
the  vicinity  of  this  contact  and  subsequently  requested  an  additional  helo  added.  REG 
queried  the  HAL  assessment  of  2275,  since  there  had  been  no  change  in  course  or 
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speed  of  the  contact  in  response  to  the  torpedo  attack.  Of  interest,  there  is  no 
immediate  reaction  from  HAL  and  it  takes  a  further  eighteen  minutes  before  this 
contact  is  re-assessed. 

•  During  the  following  1 1  minutes,  the  ORO  ranged  in  from  256  to  16  NM  for  fifteen 
seconds,  presumably  to  focus  on  the  subsurface  contact,  then  ranged  out  again  to 
128NM  to  watch  ongoing  air  contacts. 

•  At  approximately  10:1 1  ALG  helo  reported  a  subsurface  contact  from  a  sonobuoy 
dropped  prior  to  the  first  contact,  classified  as  "poss  sub  low  1". 

This  contact,  Track  2304,  is  incorrectly  entered  into  system  at  the  same  position  as  the  reporting 
helo  rather  than  at  the  bearing  and  position  reported  by  the  helo.  A  discussion  between  the  ASPO 
and  the  ASWD  some  three  minutes  later  resulted  in  this  track  plotting  error  being  rectified. 

•  At  10:15  a  new  subsurface  contact,  Track  2463,  appeared  as  a  Link  track  from  REG. 

•  By  10:16  Track  2463  had  been  classified  as  an  Akula  class  sub  and  in  response,  REG 
conducted  an  urgent  torpedo  attack. 

•  Immediately  following,  Track  2463  was  made  hostile  by  REG  and  classified  as  "poss  sub 
low  1". 

•  Shortly  thereafter  REG  reported  holding  blade  (sonar  operator  hears  propeller)  on 
Track  2463  providing  additional  evidence  that  it  was  a  sub. 

•  At  10:16,  Halifax’s  helo  reported  no  Madman  (no  Magnetic  Anomaly  Detection)  when 
over  Track  2275,  which  does  not  support  the  assessment  of  2275  as  a  sub  and  is 
consistent  with  the  earlier  questioning  by  REG  of  the  "poss  sub"  classification. 

•  At  10:17  REG  detected  a  torpedo  and  the  bearing  line  suggests  that  is  associated  with 
Akula  hostile  sub  2463.  The  situation  depicted  on  the  ASPO  display  at  this  time  is 
shown  in  Figure  5. 

•  Somewhere  between  10:17  and  10:19  hostile  Track  2463  was  changed  to  Track  2467 
while  the  toipedo  was  assigned  bearing  line  2463.  The  ASWD  immediately  provided  a 
SITREP  to  the  ORO  with  regards  to  Tracks  2275  and  2463,  which  lasted 
approximately  50  seconds. 

•  At  10:18.18  REG  reassessed  Track  2467  from  "poss  sub  low  1"  to  "prob  sub"  based 
on  the  toipedo. 

•  At  10:19,  Halifax’s  helo  reported  Madman  (detects  metal)  when  over  hostile  Track 
2467,  providing  still  further  evidence  that  it  is  a  sub.  At  about  this  time,  surface 
contacts  (7607,7610)  appear  and  consequently,  the  ORO  must  distribute  his  attention 
between  building  the  surface  and  subsurface  picture. 

•  At  10:19  Halifax  correctly  reassessed  Track  2275  as  a  decoy  and  changed  it  to  Track 
2310,  classified  as  a  hostile  non-sub. 

•  At  10:20,  as  a  result,  Track  2467  is  reassigned  as  Track  2275,  the  contact  initially 
believed  to  be  a  hostile  sub. 

In  summary,  the  general  picture  that  emerges  from  this  series  of  events  is  one  in  which  there  is 
greater  uncertainty  concerning  both  target  identity  and  location  compared  with  the  air  picture. 

There  appears  to  be  a  tendency  to  initially  classify  sub-surface  unknowns  as  possible  hostile,  while 
more  data  are  gathered  to  verify  the  initial  classification.  The  record  shows  that  picture  building, 
contact  resolution  and  threat  assessment  are  ongoing  processes  that  result  in  the  integration  of  more 
information  over  time  concerning  contacts  which  leads  to  the  refinement  of  the  identification. 

Further,  the  time  and  pace  with  which  contacts  are  processed  is  clearly  slower  than  that  of  air 
warfare.  The  gathering  of  contact  information  is  clearly  a  team  process  with  contributing 
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information  coming  from  deployed  helos,  other  ships  and  sonabuoys.  Many  of  the 
communications  that  support  this  process  take  place  on  external  circuits. 
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Figure  4.4:  Gantt  chart  showing  main  events  and  actions  for  subsurface  contacts 
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Requester  DEBRIEF  Own  ship  1  Source:  RT2  DataOime  21/6/2000  10:16:49  082165759 


Figure  4.5:  Screen  capture  of  ASPO  (RT2)  CCS  display  at  10:17  in  scenario 


4.5.2. 1  Analysis  of  ORO  sub-surface  contact  activities 

Although  the  ORO  is  active  dealing  with  above  water  threats,  there  is  little  in  the  CCS  data  record 
to  indicate  what  the  ORO  is  doing  about  the  sub-surface  threat  during  this  series  of  events.  There 
is  no  evidence  that  the  ORO  hooks  any  of  the  sub-surface  contacts,  discusses  the  situation  with  the 
ASWD  or  CO,  except  for  one  occasion  at  seventeen  minutes  into  the  scenario,  when  the  SWC 
gives  the  ORO  a  lengthy  SITREP  concerning  two  suspicious  contacts.  It  should  be  remembered 
that  overlapping  with  this  series  of  sub-surface  events  are  a  number  of  surface  contacts.  More 
analysis  of  the  communication  records  would  be  required  to  establish  exactly  what  the  ORO  was 
doing  during  these  events. 
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4.5.2.2  MOPs 

Several  relevant  MOPs  were  able  to  be  extracted  from  this  series  of  events  and  are  shown  in  the 
following  table.  For  simplicity  and  clarity  only  those  tasks  and  MOPs  relevant  to  the  sub-surface 
action  are  presented,  rather  than  reproducing  the  MOP  table  in  entirety.  To  the  level  that  we  were 
able  to  analyse  the  data  record  (including  ORO  comms  and  CCS  action)  there  were  insufficient 
data  to  generate  any  ORO  centred  MOPs.  Flence  the  examples  provided  below  are  more  relevant 
to  the  detect-to-resolve  process 


DETAILED  MEASURES  EXAMPLE  from  SCENARIO 

COMMENTS 

TASK  4.1  Identify  friendly  sub-surface 

4.1a  Accuracy  in  identifying  friendly 

sub-surface 

Correct  ID  of  location  of  friendly  bullpen. 

Team  created  bullpen  (defined  geo 
area  in  which  a  friendly  sub  is 
operating)  and  discussed  implications 
of  this  during  picture  compilation. 

(From  notes  and  observation  of 

CCS). 

TASK  4.2  Identify  hostile/suspect  sub-surface 

4.2a  Accuracy  in  identifying  hostile  / 
suspect  sub-surface 

2275:  Although  this  is  finally  assessed  correctly, 
there  are  several  incorrect  assessments  in  the 
process  of  gathering  more  information. 

2304:  Correct  ID 

2463:  Correct  ID 

4.2b  Mean  time  to  identify  hostile/suspect 
sub-surface 

2275:  RT-poss  sub:25sec 

2275:  RT-reassess  pos  sub  low2=162  sec 

2275:  RT-final  assesment=1935  sec 

2304:  RT  prb  sub=254  sec 

2463/7:  RT  hostile=51  sec 

There  is  continuing  re-assessment  of 
this  contact. 

TASK  4.3  Identify  neutral  sub-surface 

No  examples  in  scenario 

TASK  4.9  Manage  tracks 

Accuracy  in  managing  tracks 

One  example  of  poor  track  management. 

The  decoy  (track  2275)  should  have  maintained 
that  track  number  throughout.  For  some  reason  it 
was  reassigned  track  number  2310  even  after  it 
was  correctly  ID  as  a  decoy.  The  AKULA  was 
initially  2463  and  should  have  stayed  as  this.  It 
was  changed  first  to  2467  and  then  to  2275,  which 
was  really  the  decoy's  track  number. 

Post  event  analysis  by  SME 

Time  required  to  manage  tracks 

TASK  10  Manage  ship  surveillance 

10a  Accuracy  recognising  problems  in 
detect  to  resolve  process 

ORO  fails  to  see  error  in  plotting  track  2304 
(reported  from  helo).  Error  is  corrected  by  ASWD. 

10:14.04 

10b  Time  to  recognise  problems  in  detect- 
to-resolve  process 

Above  example  takes  30  sec  to  correct 

10:14.44 

Table  4.5:  Examples  of  Detect  to  Resolve  MOPs  (sub-surface  contacts)  collected 

during  Pilot  Trial 
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4.5.3  Surface  event  series:  description  and  overview 

This  series  of  events  occurs  after  the  preceding  air  series  and  runs  from  approximately  1 0: 1 8  to 
10:36. 

A  GANTT  style  chart  (Figure  6)  shows  the  relationship  among  the  various  main  events  and  actions 
for  surface  contacts. 

Prior  to  the  appearance  of  surface  contacts,  the  underwater  warfare  team  has  been  engaged  in 
underwater  picture  building  for  approximately  20  minutes.  At  the  time  the  first  surface  contact 
appears,  the  ORO  and  underwater  warfare  team  are  tracking  an  Akula  class  hostile  sub  (Track 
2275)  that  has  recently  fired  a  torpedo  at  Flalifax.  Figure  7  presents  a  screen  capture  of  the  ASPO 
CCS  display  at  10: 16:49  showing  an  area  of  probability  (AOP)  associated  with  Track  2275.  Then: 

•  At  approximately  10:18  surface  contacts,  Track  7610  and  7607,  appeared.  7610  was 
made  suspect  by  the  ASWD  before  10:19  based  on  its  proximity  to  the  location  where 
the  two  CAP  aircraft  had  been  shot  down  enroute  to  intercepting  the  Mainstay,  and 
unassociated  ESM. 

•  At  10:20,  Track  7607  is  made  suspect  based  on  its  relationship  to  the  AOP  of  one  of 
the  Surface  Action  Groups  (SAG),  however,  by  10:21  7607  is  reassessed  as  assumed 
friend  based  on  its  radar  profile. 

•  At  about  the  same  time,  two  more  surface  tracks,  7612/3  appeared  and  were  made 
suspect  almost  at  once. 

•  Just  past  10:22,  the  Commander  Task  Group  (CTG)  directs  the  MPA  to  issue  a  verbal 
warning  to  7612/3,  which  are  assessed  probable  Surface  Action  Group  (SAG)  based  on 
ESM.  Immediately  after,  two  additional  surface  contacts,  Track  7614  (30  nm  to  the 
south)  and  7615  (40nm  to  the  north)  appeared. 

At  this  point  the  ORO  is  ranged  out  suggesting  that  he  has  not  been  following  these  events  closely, 
however,  at  about  10:24  the  ORO  ranges  down  to  8NM,  although  the  reason  for  this  cannot  be 
readily  determined. 

•  Just  before  10:25,  it  is  recommended  that  Tracks  7614/5  is  made  suspect  due  to 
proximity  to  SAG  IB  AOP.  The  ASuWC  (AS)  responds  agreeing  that  7615  is  suspect 
but  recommended  7614  remain  unknown  as  the  MPA  assessed  it  too  small  for  a 
warship. 

•  At  10:25,  REG  makes  Track  7610  hostile  based  on  associated  ESM,  proximity  to 
downed  aircraft  and  behaviour.  However,  Vancouver  recommends  to  AS  that  7610  is 
outside  weapons  range  and  therefore  no  warnings  should  be  issued  by  the  MPA. 

•  At  10:26,  the  MPA  reports  that  7614/5  are  non  warships  based  on  their  radar  profile, 
and  as  a  result,  AS  recommends  that  7614/5  are  classified  assumed  friend. 

•  At  about  10:28  MPA  reports  that  7612/3  did  not  respond  to  verbal  warning  given  six 
minutes  prior. 

•  At  10:33  new  surface  tracks  7620/21/22/23  are  received  from  MPA.  Although  there  is 
no  other  evidence  suggesting  that  these  are  possible  warships,  Regina  recommends  to 
Vancouver  that  7620/21/22/23  are  made  suspect  based  on  area  of  probability  of  SAG 
IB,  and  Vancouver  agrees. 

•  Just  past  10:33  two  incoming  missiles,  Tracks  2320/1  (local  tracks  8318/9)  appear  on 
the  SWC  display.  (These  may  be  missiles  from  7612/3  given  that  they  did  not  respond 
to  verbal  warning  by  the  MPA.) 
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•  At  10:34,  Regina’s  helicopter  reports  that  suspect  Track  7620  is  a  civilian  merchant 
ship  based  on  its  radar  profile  and  is  therefore  made  assumed  friend  by  10:35. 
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Figure  4.6:  Gantt  chart  showing  main  events  and  actions  for  surface  contacts 
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4. 5. 3. 1  Analysis  of  OROs  surface  contact  activities 

There  are  very  few  communications  and  ORO  actions  to  indicate  that  the  ORO  is  tracking  these 
surface  contact  activities  closely.  The  ORO  ranges  down  only  once  during  this  period  for  an 
unknown  reason. 

4.5.3.2  MOPs 

Again,  we  have  selected  relevant  examples  of  MOPs  for  the  detect-to-resolve  process  for  surface 
contacts.  There  are  no  ORO  centred  MOPs  that  can  be  extracted  without  much  deeper  analysis  of  the 
original  scenario  record. 


DETAILED  MEASURES 

EXAMPLE  from  SCENARIO 

COMMENTS 

TASK  4.1  Identify  friendly  surface 

Did  not  capture  this  event  in 
scenario 

TASK  4.2  Identify  hostile/suspect  surface 

4.2a  Accuracy  in  identifying  hostile  / 
suspect  surface 

Two  examples  correctly  identified 

4.2b  Mean  time  to  identify  hostile/suspect 
surface 

7610:  RT  assumed  sus=  133  sec 

7610:  RT  hostile=  420  sec 

7612:  RT  assumed  sus=  40  sec 

7612:  RT  assumed  prob  sus=  90  sec 

TASK  4.3  Identify  assumed 

friendly  surface 

4.3a  Accuracy  in  identifying  friendly 

surface 

Three  examples  all  correctly  identified 

Each  initially  made  suspect 

4.3b  Mean  time  spent  in  identifying 

friendly  surface 

7607:  RT  assumed  friend=  176  sec 

7615:  RT  assumed  friend=  166  sec 

7612:  RT  assumed  friend=  165  sec 

Initially  made  suspect  (133  sec)pending 
further  evidence 

Initially  made  suspect  (50sec)  pending 
further  evidence 

Initially  made  suspect  (15  sec)  pending 
further  evidence 

Table  4.6:  Examples  of  Detect  to  Resolve  MOPs  (surface  contacts)  collected 

during  Pilot  Trial 

4.5.4  Summary  of  MOPs  that  may  be  extracted  from  the  ORTT  data  record 

Following  this  analysis  of  a  representative  data  record,  its  overall  potential  for  extraction  of  data 
pertinent  to  the  full  list  of  possible  MOPs  has  become  clearer.  To  this  end,  the  full  list  of  T&E 
MOPs  developed  in  an  earlier  report  (Matthews,  Webb  &  Keeble,  2001)  has  been  revisited  with  a 
view  to  providing  a  commentary  on  what  is  feasible  and  practicable  using  the  ORTT  data  record 
and  Debrief  function. 
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DETAILED  MEASURES 

STATUS 

COMMENTS 

T&E  TASK  1  Check  mission  picture 

la  Accuracy  in  detecting  relevant  info  within 
incoming  message  stream 

Air  example  provided 

We  did  not  request  or  have  access  to  paper  based  messages;  associated 

MOPs  based  on  paper  message  flow  were  given  zero  priority  because  they  are 
unlikely  to  be  a  source  of  useful  data  in  evaluating  the  impact  of  MSDF 
technology.  However,  it  should  be  remembered  from  the  CTA  that  these  are  an 
important  source  of  workload  for  the  ORO  and  should  be  examined  in  a  full 

T&E  trial. 

1  b  Accuracy  in  ignoring  irrelevant  info 

Not  investigated 

Feasible,  but  given  the  labour  required  to  collect  a  large  enough  database  of 
communications  for  reliable  data,  this  should  probably  have  a  lower  priority. 

1c  Total  time  spent  in  comms  dealing  with 
incoming  info 

Not  investigated 

This  is  feasible.  Given  the  labour  intensive  nature  of  this  task,  it  could  not  be 
undertaken  within  the  scope  of  the  pilot  trial.  Although,  we  believe  this  to  be  an 
important  measure  that  should  be  computed  in  future  trials. 

Id  Time  to  detect  high  salience  message 

Air  example  provided 

1e  Accuracy  in  requesting  additional  info  (i.e. 
number  of  messages  that  required  follow  up 
that  resulted  in  ORO  request  for  more  info) 

Not  investigated 

This  is  feasible,  but  given  the  labour  required  to  collect  a  sufficiently  large 
database  of  communications  to  establish  reliable  data,  this  should  probably  be 
assigned  a  lower  priority,  given  that  1c  is  likely  to  provide  a  good  general 
measure. 

If  Accuracy  in  ignoring  lower  priority  messages 

Example  provided 

Not  done  in  training  scenario  (only  at  sea)  -  may  happen  by  chance  in  ORTT, 
as  was  observed. 

T&E  TASK  2.  Relate  new  info  Ops  Room  picture 

2a  Accuracy  in  directing  (communicates) 
info/direction  resulting  from  message 

Air  example  provided 

2b  Accuracy  re  message  content  in  briefing 
appropriate  station 

Not  investigated 

2c  Accuracy  in  comprehension  of  impact  on  pre¬ 
plans/response  options/tactical  situation 

Air  example  provided 

2d  Accuracy  in  recognition  of  impact  on  Ops 

Room  capability 

N/A 

No  opportunities.  May  not  be  possible  except  at  sea  when  capabilities  become 
degraded.  This  is  not  normally  done  in  land  based  training. 

2e  Total  time  spent  in  relaying  info  re  Ops  Room 
status 

Not  investigated 

Feasible,  but  time  consuming.  Suggest  lower  priority  for  future  trials 

T&E  TASK  3  Relate  (new  info)  to  RMP,  RAP 

Includes  aspects  of  build  awareness  across  domains  (air,  surface,  sub-surface). 

3a  Accuracy  of  salient  info  within  RMP  prior  to 
new  message 

Not  investigated 

Not  feasible  unless  ORO  provides  a  timely  SITREP  as  conditions  demand. 

3b  Accuracy  of  salient  info  within  RMP  after 
new  message 

Not  investigated 

Feasible  if  we  infer  ORO  subsequent  actions  are  based  on  new  info. 

3c  Time  for  ORO  to  complete  understanding  of 
new  info  regarding  RMP 

Not  investigated 

Feasible  if  we  infer  ORO  subsequent  actions  are  based  on  new  info.  However, 
comprehension  may  not  necessarily  lead  to  a  specific,  observable  ORO 
behaviour. 

3d  Accuracy  in  taking  appropriate  action  as  a 
result  of  updated  RMP 

Air  example  provided 

3e  Total  time  spent  on  SSD  relating  new  info  to 
RMP 

Not  investigated 

This  may  be  too  complex  to  assemble,  since  start  and  end  points  are  not 
necessarily  definable. 

3f  Accuracy  in  assessing  the  integrated  tactical 
picture 

Example  provided 

Table  4.7:  Summary  of  MOPs  potentially  collectible  in  the  ORTT  based  upon  pilot 

trial  (continued) 
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DETAILED  MEASURES 

STATUS 

COMMENTS 

T&E  TASK  4  Relate  new  info  to  air,  surface,  sub-surface  picture 

SUB  TASK  4.1  Identify  friendly  contact  (air,  surface,  sub-surface) 

4.1a  Accuracy  in  identifying  friendly  contact 

Air,  surface 
examples  provided 

4.1b  Mean  time  spent  in  identifying  friendly 
contact 

Air,  surface 
examples  provided 

4.1c  Total  number  of  queries  or  total  time 
spent  in  querying  team  for  additional 
info 

Not  investigated 

Feasible.  Requires  time  consuming  analysis  of  comms 

SUB  TASK  4.2  Identify  hostile/suspect  contact 

4.2a  Accuracy  in  identifying  hostile  /  suspect 
contact 

Air,  surface,  sub¬ 
surface  examples 
provided 

4.2b  Mean  time  to  identify  hostile/suspect 
contact 

Air,  surface,  sub¬ 
surface  examples 
provided 

4.2c  Total  number  of  queries  or  total  time 
spent  in  querying  team  for  additional 
info 

Air  example 
provided 

SUB  TASK  4.3 

Identify  neutral  contact 

Not  investigated 

Feasible  based  upon  methodology  in  4.2 

SUB  TASK  4.4 

Identify  NU  tracks  (non-updated) 

Not  investigated 

Not  normally  included  in  scenario  in  ORTT 

SUB  TASK  4.5 

Identify  tracks  reported  by  ownship  or, 
conversely,  by  other  participating  units 
(PUS) 

Not  investigated 

Possibly  feasible;  information  might  be  obtained  from  more  in- 
depth  analysis  of  comms,  but  would  likely  require  debrief  of 
team. 

SUB  TASK  4.6 

Identify  own  force  engagement  status 

4.6a  Accuracy  identifying  air  tracks  being 
engaged  by  ownship  or  ships  in  TG 

Not  investigated 

Needs  a  T&E  probe-probably  not  available 

4.6b  Response  time  to  be  ready  to  engage 
correct  track 

Not  investigated 

Feasible 

4.6c  Total  number  of  queries  or  total  time 
spent  in  querying  team  for  info 

Not  investigated 

Feasible.  Requires  time  consuming  analysis  of  comms 

Table  4.7:  Summary  of  MOPs  potentially  collectible  in  the  ORTT 
based  upon  pilot  trial  (continued) 
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DETAILED  MEASURES 

STATUS 

COMMENTS 

SUB  TASK  4.7  React  to  threat  track  symbology  LINKed  by 

consort  not  on  video 

4.7a  Time  to  recognise  symbology  not  on 
video 

Not  investigated 

N/A  in  ORTT 

4.7b  Time  to  order  remedial  action 

Not  investigated 

N/A  in  ORTT 

SUB  TASK  4.8  React  to  LINK  not  gridlocked 

N/A  in  ORTT 

4.8a  Time  to  recognise  LINK  not  gridlocked 

N/A  in  ORTT 

4.8b  Time  to  order  remedial  action 

N/A  in  ORTT 

SUB  TASK  4.9  Resolve  ambiguous/dual  tracks 

4.9a  Accuracy  in  resolving  dual  tracks 

Air  example 
provided 

4.9b  Time  required  to  resolve  dual  tracks 

Air  example 
provided 

4.9c  Total  number  of  queries  or  total  time 
spent  in  querying  team  for  info 

Not  investigated 

Feasible.  Requires  time  consuming  analysis  of  comms  but  is  a 
desirable  measure  to  be  collected  in  future  trials. 

T&E  TASK  7  Assess  threats  -generic 

SUB  TASK  7.1  Detect  changes  in  tactical  situation 

Evidence  from  the  scenario  suggests  that  measures  7. la-7. Id 
can  all  be  considered  as  part  of  threat  identification  in  the 
detect  to  resolve  cycle. 

7.1a  Accuracy  in  ID  new  threats 

Examples  provided: 
air,  surface,  sub¬ 
surface 

7.1b  Accuracy  in  ID  changes  in  threat  status 

Air,  surface,  sub¬ 
surface  examples 
provided 

7.1c  Time  to  detect  new  threat 

Air,  surface,  sub¬ 
surface  examples 
provided 

7.1  d  Time  to  detect  change  in  threat  status 

Air,  surface,  sub¬ 
surface  examples 
provided 

7.1  e  Time  to  provide  SITREP  in  response  to 
request 

Not  investigated 

No  example  in  scenario.  In  general,  it  is  not  possible  to 
determine  start  point  of  SITREP  creation  without  a  probe  or 
unless  the  CO  requests  one. 

7. If.  Accuracy  in  SITREP  contents 

Not  investigated 

Feasible:  will  require  post-event  analysis  by  SME 

SUB  TASK  7.2  Determine  threat  priorities  across  domains 

7.2a  Accuracy  in  ranking  threat  priorities 
across  domains 

Example  provided 

Post-event  analysis  by  SME  indicated  some  surprise  over  the 
apparently  low  priority  the  ORO  (and  the  CO)  gave  the  ASW 
threat  as  it  was  building,  given  the  lethality  of  a  torpedo  strike 
compared  with  that  of  a  missile. 

7.2b  Time  to  assess  threat  priorities  across 
domains 

Not  investigated 

It  would  be  difficult  to  determine  the  start  point  for  this  time 
measure,  without  a  probe  methodology. 

Table  4.7:  Summary  of  MOPs  potentially  collectible  in  the  ORTT 
based  upon  pilot  trial  (continued) 
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DETAILED  MEASURES 

STATUS 

COMMENTS 

SUB  TASK  7.3  Assess  threats-within  a  specific  warfare 

domain.  Identify  track  threat  priority 

7.3a  Accuracy  in  identifying  threat  priorities 

Air  example 

7.3b  Total  time  to  identify  highest  priority 
threats 

Not  investigated 

This  is  probably  not  a  feasible  approach  in  the  ORTT. 

7.3c  Time  to  annotate  CCS  with  CPA 

No  evidence  done 
for  air  tracks 

Feasible,  but  7.3e  may  be  a  better  MOP  for  this  process  of 
assessment. 

7.3d  Accuracy  in  determining  CPA 

No  evidence  done 
for  air  tracks 

Feasible,  but  7.3f  may  be  a  better  MOP  for  this  process  of 
assessment 

7.3e  Accuracy  in  determining  lethality 

Air  example 

Possible,  given  appropriate  scenario  circumstances 

7.3f  Time  to  determine  lethality 

Not  investigated 

Possible,  given  appropriate  scenario  circumstances 

7.3g  Time  to  create  a  SITREP 

Not  investigated 

Not  possible  to  determine  start  point  of  SITREP  creation 
without  a  probe  or  unless  the  CO  requests  one. 

7.3h  Accuracy  and  completeness  of  SITREP 

Feasible:  will  require  analysis  by  SME 

SUB  TASK  7.4  Analyse  history 

profile  of  hostile 
surface 

Not  investigated 

Not  relevant;  no  repeated  action  in  scenarios 

T&E  TASK  9  Assess  sensors: 

Not  investigated 

Not  feasible  in  existing  scenarios 

T&E  TASK  10  Manage  ship  surveillance 

10a  Accuracy  recognising  problems  in 
detect  to  resolve  process 

Air,  sub-surface 
examples 

10b  Time  to  recognise  problems  in  detect- 
to-resolve  process 

Air,  sub-surface 
examples 

1 0c  Appropriateness  of  remedial  action  to 
correct  errors 

Air,  sub-surface 
examples 

Table  4.7:  Summary  of  MOPs  potentially  collectible  in  the  ORTT 

based  upon  pilot  trial 

To  summarise,  of  these  sixty  potential  MOPs,  we  have  been  able  to  provide  concrete  examples  for 
nineteen  and  believe  that  a  further  eleven  are  feasible,  given  appropriate  scenario  circumstances 
and/or  time  for  more  in-depth  analysis.  Of  significance  is  the  fact  that  we  are  able  to  extract 
quantifiable  MOPs  for  the  following  ORO  core  functions  (based  upon  the  CTA): 
building/maintaining  the  global,  mission  and  tactical  pictures,  managing  ship  surveillance, 
managing  ship's  response  (including  contact  classification).  Other  sub-functions  such  as  the 
assessment  of  communications  and  weapons  would  be  amenable  to  MOP  data  extraction,  if  the 
scenario  circumstances  provide  the  appropriate  context.  The  only  areas  where  it  does  not  seem 
feasible  to  extract  MOP  data  from  the  ORTT  record  is  for  "assess  teams",  "assess  schedules"  and 
"assess  sensors"  (sub-functions  of  "managing  ship's  capability"). 

Of  importance  to  the  future  evaluation  of  COMDAT  MSDF  technology,  we  have  clearly 
demonstrated  that  the  ORTT  data  record  provides  us  with  a  capability  to  capture  a  number  of 
MOPs  that  relate  to  the  critical  processes  of  the  detect-to-resolve  cycle  and  picture  compilation  and 
associated  communications. 
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4.5.5  Recommendations  for  subset  of  MOPs  to  be  used  for  assessing  COMDAT 
MSDF  in  upcoming  trials 

As  shown  in  the  above  section,  many  MOPs  may  be  feasibly  collected  in  the  ORTT  that  are 
directly  related  to  Ops  Room  and  ORO  critical  functions.  Some  of  these  MOPs  are  more  relevant 
to  MSDF  evaluation  than  others,  some  are  more  feasible  to  collect  than  others,  and  amongst  those, 
some  are  more  labour  intensive  to  collect  than  others.  Having  assessed  each  of  these  issues  for 
each  MOP  in  turn,  we  believe  that  the  following  subset  of  MOPs  may  be  the  most  diagnostic  and 
practical  for  assessing  MSDF  technology  i.e.  the  functions  these  MOPs  represent  should  benefit 
from  MSDF  technology.  For  example,  with  MSDF,  operators  should  be  more  confident  and 
possibly  take  less  time  to  identify  and  track  contacts,  require  fewer  communications  to  resolve 
contact  ambiguities  than  at  present,  and  spend  less  time  micro-managing  their  sensors  to  resolve 
track  ambiguities  and  the  like.  The  outcome  of  this  should  be  more  time  spent  assessing  the 
tactical  implications  and  more  effective  within  and  between  team  communication  about  tactical 
rather  than  contact  and  track  management  issues. 


MOP  DESCRIPTION 

DETAILS 

COMMENT 

Accuracy  In  assessing  the  Integrated 
tactical  picture 

Includes:  multi-sensor/source  and  wide 
area  picture  integration,  surface/sub¬ 
surface  picture  compilation 

Total  time  in  communication 

Sum  total  of  time  in  net-based  comms 

Total  communications  across  team,  TG  and 
other  sources  of  information 

Communication  errors 

Percent  of  comm  errors  as  a  function  of 
total  number  of  comms  per  event 

Communication  errors  can  be  semantic, 
requests  for  repetition,  untimely,  missed. 

Wide  area  picture  quality  compared 
with  ground  truth  and/or  information 
available  to  the  Ops  Room 

Navy  SME  judgments  on  behaviourally 
anchored  rating  scales. 

Scales  to  be  developed  by  SMEs  prior  to 
trial. 

Identify  hostile/suspect  contact 

These  can  be  applied  to  air,  surface  and 
sub-surface  contacts 

Accuracy 

Percent  correct  ID 

May  also  be  expressed  as  error  rate 

Response  time  per  contact 

Average  time  from  triggering  event  to 
interim  or  final  ID. 

Total  time  in  communication 

Sum  total  of  time  in  net-based  comms 

Total  communications  across  team 

Communication  errors 

Percent  of  comm  errors  as  a  function  of 
total  number  of  comms  per  event 

Communication  errors  can  be  semantic, 
requests  for  repetition,  untimely,  missed. 

Manage  ambiguous  tracks 

Covers  all  aspects  of  ambiguous  tracking 
whether  from  sensor  degradation  to 
multiple  sensors  holding  same  contact  and 
reporting/comm  errors  on  tracks 

Accuracy  in  resolving  dual  tracks 

Percent  of  dual  tracking  situations  that  are 
correctly  resolved 

Time  required 

Average  time  per  dual  track  situation  before 
correct  solution  is  obtained 

Total  time  in  communication 

Sum  total  of  time  in  net-based  comms 
related  to  resolving  tracks 

Manage  ship  surveillance 

ORO  centred  measures 

ORO  accuracy  in  recognising 
problems  in  detect  to  resolve  process 

Number  of  correct  IDs  by  ORO  divided  by 
number  of  opportunities 

May  be  few  of  these  with  experienced  team 

Time  to  recognise  problems  in  detect- 
to-resolve  process 

Time  from  first  indication  of  problem  until 
resolution 

Table  4.8:  Recommended  MOPs  for  assessing  MSDF  impact 
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4.5.6  The  efficiency  of  the  data  extraction  process 

4.5.6. 1  ORTT  CCS  and  Communication  data  record 

The  pilot  trial  has  provided  an  opportunity  to  explore  the  ways  in  which  the  ORTT  data  record  may 
be  analysed  for  the  purposes  of  extracting  MOP  data.  The  experience  of  doing  this  and  learning 
effective  and  less  effective  approaches  has  led  us  to  a  better  appreciation  of  the  time  required  to 
extract  data  of  interest.  Generally  speaking,  the  CCS  record  may  be  analysed  more  efficiently  than 
the  communication  record  for  a  number  of  reasons.  First,  we  can  view  up  to  three  CCS  consoles 
simultaneously  during  replay,  whereas  we  can  only  monitor  one  communication  net.  Second,  the 
CCS  record  is  relatively  slow  moving  and  allows  the  information  to  be  extracted  with  fewer  passes 
through  the  same  segment.  In  contrast,  it  may  take  a  number  of  replays  of  quickly  spoken 
messages  to  get  the  content.  Third,  to  fully  track  down  the  flow  of  information  in  complex 
situations,  it  may  be  necessary  to  examine  several  communication  nets  using  many  iterative  steps. 

There  are  three  stages  to  the  data  analysis  process. 

•  The  first  is  a  re-creation  of  the  critical  scenario  events  along  a  common  time  base. 

This  includes  all  relevant  communications  (source,  destination,  content),  CCS  contacts 
(location,  radar  held,  status),  changes  in  contact  status,  ORO  range  and  hook  selections 
(possibly  also  those  of  the  SWC  and  ARRO,  ASPO)  and  any  contextual  explanatory 
notes  for  each  entry  from  the  T&E  team.  Based  upon  our  first  attempt  at  this  process 
during  the  T&E  trial  it  took  approximately  twelve  hours  for  three  persons  to  re-create 
approximately  90  minutes  of  scenario.  Part  of  this  time  was  spent  in  reviewing  the 
whole  scenario  for  the  events  to  be  re-created  and  part  in  figuring  out  a  suitable 
methodology.  It  should  be  noted  that  real-time  access  to  the  ORTT  playback  tools  and 
data  record  is  required  to  conduct  this  stage. 

•  The  second  stage  is  a  "semantic"  analysis  of  the  output  from  the  first  stage  to 
comprehend  the  chain  of  events  and  the  Ops  Room  team  behaviours. 

•  The  third  stage  is  the  extraction  of  the  relevant  MOPs  using  the  products  of  the  first 
two  stages.  These  second  and  third  stages  have  taken  approximately  thirty  person 
hours  and  have  been  conducted  off-site  from  the  ORTT. 

With  the  benefit  of  lessons  learned  from  this  pilot  trial,  greater  efficiencies  in  these  processes  will 
evolve,  especially  if  the  goal  is  to  concentrate  on  a  subset  of  measures  rather  than  trying  to  evaluate 
the  overall  potential  of  the  ORTT  to  support  all  of  the  MOPs  that  have  been  initially  identified. 

4. 5. 6. 2  Keystroke  file 

We  conducted  a  brief  review  of  the  keystroke  file  in  order  to  assess  its  usefulness  in  extracting 
some  micro-level  timing  data.  In  its  present  format,  it  does  not  facilitate  rapid  data  extraction,  as 
the  record  has  to  be  scanned  sequentially  to  find  data  of  interest.  For  security  reasons,  the  data  may 
not  be  printed  out  and  there  is  no  software  to  translate  the  record  into  one  that  is  compatible  with  a 
spreadsheet  format.  Should  such  a  capability  be  made  available  or  created,  then  the  simple 
spreadsheet  macros  should  allow  the  rapid  extraction  of  important  data  related  to  designated  keys 
or  keystroke  sequences.  An  example  of  this  might  be  the  elapsed  time  between  the  ARRO  entering 
a  track  number  in  the  system  and  the  time  when  that  track  is  later  correlated  with  other  data  such  as 
EW. 
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4. 5. 6. 3  Audio/video  tape  record 

This  medium  provides  a  audio-visual  data  record  of  the  interactions  between  the  ORO  and  other 
members  of  the  team  in  the  Ops  room,  as  well  as  the  associated  face  to  face  conversations  that  are 
not  recorded  over  a  circuit.  We  did  not  find  a  need  during  the  analysis  of  the  data  record  to  consult 
this  tape. 

Based  upon  what  we  have  learned,  we  believe  that  this  record  may  only  need  to  be  accessed  at 
specific  times  when  critical  information  appears  to  have  been  passed  outside  of  the  electronic 
communication  nets.  As  far  as  we  could  tell,  the  scenario  events  we  saw  neither  provided  a  large 
body  of  text  messages  that  were  hand  delivered  to  the  ORO  nor  a  need  for  face  to  face  discussions 
among  the  team.  However,  it  should  be  acknowledged  that  we  did  not  give  high  priority  to  find 
evidence  of  such  communication  given  the  analysis  time  available  on  site.  Furthermore  the  reports 
of  actual  operations  at  sea  suggest  that,  the  volume  and  significance  of  text  messages  at  least,  is  far 
greater  than  that  simulated  in  the  ORTT. 

Thus,  it  may  have  been  possible  that  the  ORS  was  busy  communicating  face  to  face  with  many 
team  members,  but  we  didn’t  see  this  because  we  weren’t  looking.  It  certainly  wasn’t  evident  from 
the  ORTT  record,  but  it  may  have  happened.  Notwithstanding  this  possibility,  it  does  seem  that 
this  scenario  did  not  place  as  high  a  requirement  for  face  to  face  comms  as  a  real  scenario  at  sea 
would.  Many  of  the  face  to  face  comms  at  sea  are  between  the  ORO/ORS  and  often  deal  with  Ops 
Room  management  issues  that  typically  do  not  arise  in  a  training  environment. 

In  general,  we  conclude  that  for  future  MOP  data  extraction,  this  data  record  will  not  have  to  be 
analysed  in  its  entirety  but  only  minimally  consulted  where  scenario  circumstances  warrant. 

4.6  Discussion 

4.6.1  Review  of  goals 

First,  we  will  review  our  success  in  meeting  the  goals  that  were  set  out  in  Section  2  of  the  report. 

•  Observe  a  training  scenario  in  action  and  note  areas  of  interest  for  MOPs 

We  were  able  to  observe  approximately  three  hours  of  training  with  an  intact  ship's  Ops  Room 
Team.  However,  the  scenario  observed  was  somewhat  different  from  the  regular  training  scenario 
in  that  it  required  the  ship  to  pursue  a  possible  hostile  contact  without  the  usual  context  of  a  TG. 
This  contrasts  with  a  typical  training  scenario  in  which  the  ship  under  training  responds  to  a  series 
of  air,  surface  and  sub-surface  contacts  and  threats  in  a  typical  TG  format.  The  more  typical 
training  scenario  was  considered  to  be  more  representative  and  appropriate  for  the  analysis  of 
MOPs  and  extraction  of  data.  Thus,  the  scenario  under  observation  did  not  allow  us  to  note 
specific  scenario  events  that  would  be  the  focus  of  MOP  analysis.  Instead,  we  used  a  record  of  a 
more  typical  TG  scenario  record  from  the  previous  day,  which  meant  that  there  was  some 
additional  analysis  overhead  in  first  reviewing  the  record  for  areas  of  interest  for  analysis. 

•  Assess  the  practicality  of  inserting  T&E  "tags"  into  the  scenario  record  during 
scenario  execution  to  assist  in  subsequent  MOP  analysis 

Because  we  were  not  present  during  scenario  execution,  we  were  not  able  to  do  this.  However,  our 
discussions  with  the  ORTT  technical  manager  suggest  that  this  would  be  easy  to  do  for  future 
studies. 
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•  Develop  a  methodology  for  analysis  and  recording  the  captured  data 

This  was  successfully  accomplished  and  will  be  refined  for  future  trials  based  on  lessons  learned. 

•  Analyse  the  digital  video  and  audio  record  (using  ORTT  Debrief  software)  to 
build  the  ongoing  team  picture  and  communications  for  selected  events 

As  the  detailed  analysis  in  the  results  section  shows,  we  were  successfully  able  to  accomplish  this 
for  air,  surface  and  sub-surface  events. 

•  Analyse  the  supplementary  video  and  audio  record  (using  standard  VCR)  to  build 
the  ongoing  team  picture  and  communications  for  selected  events 

There  was  insufficient  time  to  attempt  this  in  detail.  However,  the  analysis  of  the  data  record 
suggested  that  for  this  series  of  events  and  team  responses,  there  would  be  little  added  value  in 
doing  this  analysis. 

•  Derive  quantifiable  MOPs  from  the  previous  analyses 

A  number  of  examples  of  MOP  data  across  a  broad  spectrum  of  events  were  successfully  analysed. 

•  Determine  range  of  potential  MOPs  that  could  be  assessed  using  the  ORTT 

This  was  accomplished  and  the  results  are  shown  in  summary  format  in  Table  6. 

•  Make  recommendations  for  the  specific  subset  of  MOPs  that  would  be  most 
suitable  for  assessing  the  impact  of  MSDF  technology  in  future  trials 

Twelve  appropriate  and  logistically  practical  MOPs  have  been  identified  (Table  7) 

•  Comment  upon  the  effectiveness  and  efficiency  of  the  data  extraction  and  analysis 
methodology  and  its  implications  for  logistical  and  other  support  in  future  trials. 

Approximate  estimates  of  the  time  and  personnel  resources  required  to  extract  MOP  data  are 
provided.  Future  trials  will  benefit  from  the  experience  gained  and  the  process  will  become  more 
efficient.  Ultimately,  the  complexity  of  the  events  will  dictate  the  level  of  support  required  for 
analysis.  The  analysis  of  communications,  while  important  and  with  significant  potential  payback 
from  several  perspectives,  is  the  most  time  consuming  and  least  efficient,  because  of  the  inability  to 
replay  more  than  one  network  at  a  time.  A  technical  solution  should  be  sought  to  enhance  the 
efficiency  of  communication  analysis. 

4.6.2  General  assessment 

Based  on  this  Pilot  Trial,  we  believe  that  the  ORTT  represents  the  best  option  available  for 
collecting  MOPs  in  future  trials,  whether  the  focus  is  decision  support  for  the  ORO,  or  other 
functions  of  the  team  that  are  relevant  to  the  assessment  of  the  impact  of  MSDF  technology. 

For  the  ORO,  it  was  shown  that  it  is  possible  to  collect  MOPs  relevant  to  the  entire  detect  to 
engage  cycle: 

•  Building/maintaining  the  global  picture 

•  Building/maintaining  local  air,  surface,  sub-surface  pictures 

•  Exchanging  information  among  Ops  room  team  members 

•  Managing  ship  surveillance. 

•  Contact  detection,  identification  and  classification 
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•  Implementing  appropriate  responses  and 

•  Assessing  the  effectiveness  of  the  response  and  any  follow-on  actions  required. 

In  the  cases  of  picture  building,  the  MOPs  we  have  been  able  to  collect  are  dependent  upon 
observable  responses/actions/communications  of  the  ORO.  It  does  not  seem  feasible  within  the 
constraints  of  the  ORTT  training  environment  to  “get  inside  the  ORO’s  head”  using  a  probe 
methodology  that  requires  momentary  interruption  of  the  exercise  to  probe  the  situation  awareness 
of  selected  players.  The  exception  might  be  to  request  an  occasional  SITREP  set  in  the  context  of 
the  exercise.  However,  there  are  potential  training  benefits  that  come  with  the  interruption  probe 
approach.  The  possibility  that  use  of  the  interruption  probe  could  serve  both  training  and  T&  E 
should  be  investigated  in  conjunction  with  the  training  community. 

Areas  such  as  the  assessment  of  sensors,  weapons,  teams,  and  schedule  seem  less  amenable  to  the 
collection  of  MOP  data  in  the  ORTT  and  are  more  dependent  on  the  scenario  context  providing  the 
right  triggering  conditions  for  ORO  actions.  This  will  generally  be  true  of  any  land-based 
simulation  facility  and  hence  MOP  development  in  these  areas  may  require  trials  involving  the  ship 
at  sea. 

With  respect  to  MOPs  relevant  to  MSDF  assessment  and  the  picture  compilation  process,  we  have 
been  able  to  demonstrate  an  ability  to  collect  representative  data  in  the  detect-to-resolve  cycle  for 
air,  surface  and  sub-surface  contacts.  We  have  also  shown  that  we  can  identify,  time  and  assess  the 
accuracy  of  key  events  in  this  cycle,  such  as  the  exact  times  when  targets  are  first  detected,  when 
assessed  and  when  identified.  Thus,  it  seems  entirely  feasible  that  we  can  collect  quantitative  data, 
expressed  in  unambiguous  numerical  terms  that  will  allow  the  empirical  assessment  of  the  changes 
in  task  performance  that  are  attributable  to  MSDF  technology.  Such  MOP  data  will  comprise  for 
example  percent  correct  detections,  identifications,  error  rates,  time  spent  in  communications  and 
communication  errors. 

Finally,  it  should  be  noted  that  the  above  MOP  data  should  be  regarded  as  concrete  exemplars  to 
prove  the  concept  that  quantitative  data  can  be  extracted  from  the  ORTT  record  using  a  workable 
methodology.  The  numbers  themselves  should  not  be  taken  as  being  indicative  or  representative  of 
baseline  performance. 

4.6.3  Data  collection  to  support  future  MSDF  TD  trials 

Wherever  the  location  for  evaluating  the  MSDF  workstation  in  future  TD  trials,  there  will  be  two 
measurement  requirements.  The  first  will  be  to  assess  the  performance  of  the  primary  tasks  that  as 
they  are  impacted  by  the  MSDF  technology,  the  second  to  provide  baseline  comparative  data  for 
these  tasks  against  which  to  evaluate  the  technology. 

With  respect  to  the  former,  we  have  provided  a  set  of  MOPs  that  should  permit  the  assessment  of  a 
broad  range  of  tasks,  and  which  could  be  readily  adapted  to  generic  picture  compilation  and  contact 
resolution  tasks  that  may  be  affected  by  the  technology.  These  MOPs  should  be  appropriate  no 
matter  the  specific  implementation  of  the  workstation  functionality  that  provides  the  technology  and 
the  particular  constraints  it  might  place  upon  how  the  MOP  data  are  actually  collected.  At  a  practical 
level,  and  based  upon  previous  TD  trials,  it  can  be  expected  that  there  will  not  be  opportunities  for 
multiple  data  runs  with  a  reasonable  number  of  trial  participants.  Hence,  the  data  collected  are  likely 
to  be  constrained  and  at  best  indicative  of  performance  trends. 

It  follows  that  if  the  TD  trial  data  are  not  as  extensive  as  would  be  optimally  desired,  then  to  have 
baseline,  comparative  data  that  are  also  limited  in  terms  of  reliability  and  generality,  would  militate 
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against  the  drawing  of  meaningful  conclusions  concerning  the  merits  of  the  technology.  It  is 
suggested  therefore  that  a  stable  baseline  of  performance  data  be  collected  in  the  ORTT  across  a 
range  of  scenario  events  (from  existing  training  scenarios)  that  are  comparable  to  the  planned  events 
for  the  TD  trials.  In  this  way  we  can  at  least  establish  a  reliable  and  stable  basis  for  comparison. 

For  the  future,  it  may  be  appropriate  to  consider  the  degree  to  which  the  ORTT  software  can  be 
upgraded  to  allow  integration  of  MSDF,  or  other  emerging  technology,  features.  We  have  been  told 
that  this  may  not  be  difficult  to  accomplish  compared  with  the  level  of  effort  required  to  integrate 
such  features  into  the  CSTC,  particularly  if  reliable  MOP  data  are  to  be  collected. 

4.7  Recommendations 

1.  To  quantify  evaluation  of  MSDF  technology,  two  parallel  sets  of  MOP  data  must  be 
collected  to  reflect  performance  on  relevant  tasks  with  and  without  the  technology.  As  a 
next  step,  and  prior  to  the  conduct  of  the  MSDF  trials,  a  full  data  collection  trial  should  be 
conducted  to  generate  the  baseline  comparative  MOP  data  set  (using  the  MOPs  short-listed 
in  this  report)  for  MSDF  relevant  contexts  and  tasks.  This  would  be  conducted  using 
ORTT  records  from  ongoing  scenario  training  with  experienced  ship  teams.  Data 
collection  would  be  limited  to  the  specific  subset  of  MOPs  relevant  to  the  impact  of  MSDF 
technology. 

2.  To  facilitate  (1),  LMC  and  the  Navy  should  be  requested  to  develop  a  standalone  system 
for  the  "Debrief'  software  to  allow  it  to  function  independently  of  the  existing  ORTT 
system.  This  will  allow  parallel,  simultaneous  analysis  of  data  for  T&  E  when  the  ORTT 
is  being  used  for  training.  Doing  this  would  greatly  enhance  the  efficiency  and  cost- 
effectiveness  of  data  extraction.  If  this  is  not  feasible,  it  will  impact  only  on  the  efficiency 
and  cost  of  data  collection  and  should  not  be  though  of  as  a  pre-requisite  for  future  work. 

3.  To  facilitate  the  collection  of  data  in  the  CSTC  (should  it  be  required),  acquire,  or  make 
available,  supplementary  digital  data  capture,  such  as  the  DGx  system. 

4.  If  the  MSDF  TD  is  to  be  implemented  in  the  CSTC,  conduct  a  pilot  trial  in  this  facility  to 
fully  evaluate  the  ability  to  collect  the  required  MOPs  identified  for  the  evaluation  of  the 
impact  of  the  MSDF  technology. 

5.  Conduct  an  evaluation  of  the  capability  of  the  COMDAT  MSDF  prototype  workstation  and 
software  to  collect  MOP  data  for  test  and  evaluation  purposes. 
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5.  Technical  Memorandum: 


PROVISION  OF  NAVAL  SME  SUPPORT  TO  CONTRACTORS 

5.1  Summary 

5.1.1  Background 

In  the  course  of  projects  for  the  Navy  or  with  Navy  content,  contractors  have  a  need  for  Subject 
Matter  Experts  (SME)  for  blocks  of  time  (e.g.  several  hours  to  several  days)  to  participate  in  trials 
or  to  meet  with  contractors  to  explain  operational  procedures  and  processes.  In  some  cases, 
contractors  do  not  get  the  required  mix  of  qualifications  and  experience  from  the  Navy  personnel 
who  are  available  to  be  supplied.  Equally,  the  Navy  has  to  contend  with  potentially  uncoordinated 
requests  from  multiple  contractors.  This  Technical  Memorandum  addresses  these  problems  and 
seeks  to  find  a  way  to  lessen  demand  and  administrative  burden  whilst  still  providing  the  required 
SMEs  to  contractors. 

5.1.2  Findings 

It  is  clear  that  the  Navy  has  a  great  number  of  demands  on  the  time  of  its  personnel;  chief  among 
them  being  National  Defence.  The  longer  term  goals  of  research  and  development  may  conflict 
with  the  immediate  goals  associated  with  maintenance  of  a  potent  defence  force.  These  conflicts 
also  create  long  lead  times  for  requests  for  SMEs  that  can  also  make  it  difficult  to  accommodate 
last  minute  changes  in  research  and  development  projects. 

This  Technical  Memorandum  considered  six  options  for  the  provision  of  SMEs: 

•  Retention  of  the  current  structure  and  approach; 

•  Creation  of  a  centralised  SME  availability  pool; 

•  Attachment  of  serving  SMEs  to  a  project; 

•  Payment  of  serving  SMEs  acting  as  consultants  in  their  off-time; 

•  Creation  of  an  independent  SME  pool  of  Navy  retirees;  and 

•  A  hybrid  approach. 

The  annex  to  this  Technical  Memorandum  included  a  listing  of  the  functions  that  must  be  covered 
by  the  SME  pool: 

•  Radar  tracking/Above  water  picture  compilation; 

•  Underwater  picture  compilation; 

•  Electronic  Support  Measures  (ESM); 

•  Fire  control; 

•  Aircraft  control; 
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•  Warfare  sub-team  direction; 

•  Ops  room  direction;  and 

•  Command. 

Each  function  was  described  according  to  the  positions  it  relates  to, 
MOC/qualification/rank/experience,  responsibilities,  criticality  and  availability. 

5.1.3  Conclusions 

It  was  recommended  that  a  hybrid  approach  be  used,  consisting  of  elements  of  the  current  system, 
attaching  serving  SMEs  to  a  project,  and  paying  SMEs  in  their  off-time.  Challenges  to  this  hybrid 
approach  would  include  negotiating  the  involvement  of  a  dedicated  SME  on  a  project,  setting 
SMEs  up  as  self-employed  consultants  (or  else  paying  them  through  Navy  pay  channels),  and 
cultural  barriers  to  ‘moonlighting’. 

5.2  Introduction 

This  memorandum  addresses  potential  means  of  streamlining  or  reducing  the  demands  made  on 
DND  to  provide  Subject  Matter  Experts  (SMEs)  to  support  contractors  working  for  DRDC  on 
various  Navy  projects.  From  the  contractor’s  point  of  view,  actual  servicing  of  their  needs  is 
patchy  -  sometimes  spot  on,  but  sometimes  misunderstood  in  terms  of  the  need  for  currency  and 
focus  of  experience.  Conversely,  from  the  Navy’s  point  of  view,  contractors  as  a  group  can  appear 
to  be  making  similar,  uncoordinated,  and  last  minute  demands  on  scarce  resources. 

While  the  demand  for  SMEs  to  support  contractors  will  continue,  it  could  be  advantageous  to 
investigate  means  of  providing  the  support  with  reduced  demand  on  limited  naval  resources,  and  in 
such  a  way  as  to  lighten  the  administrative  load  of  setting  up  the  support.  The  concept  of 
developing  a  pool  of  naval  SMEs  upon  which  to  draw  was  first  considered  during  work  on  the 
COMDAT  project,  where  it  became  apparent  that  in  the  current  operational  climate  the  Navy  had 
considerable  difficulty  providing  current,  qualified  combat  operators  to  support  even  moderate 
contractor  requirements.  This  led  to  the  broader  investigation  described  in  this  memorandum  of 
various  options  for  providing  naval  SME  support  to  contractors. 

5.2.1  Definition  of  the  Problem 

Contractors  frequently  need  SMEs  to  support  the  work  they  have  been  contracted  to  do.  Because 
of  the  complexity  of  the  naval  operations  environment,  these  SMEs  must  have  had  recent 
operational  experience  using  the  particular  equipment  or  processes  being  studied.  In  the  case  of  a 
combat  team  or  sub-team,  several  SMEs  with  different  qualifications  or  experience  may  be 
required  concurrently.  Also,  to  gain  a  consensus  and  cover  a  range  of  experience  during  interviews 
or  focus  groups,  or  to  achieve  statistical  confidence  in  performance  studies,  two  or  even  more 
SMEs  with  similar  qualifications  and  years  of  experience  may  be  needed. 

Due  to  the  Navy’s  demanding  operational  schedule  and  current  personnel  levels  suitable  SME 
support  is  difficult  to  arrange.  Challenges  include  but  are  not  limited  to: 

•  Ships’  combat  operators  are  busy  training  or  planning  for  upcoming  operations,  or  are 
readjusting  to  home  port  routine  after  returning  from  deployment 
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•  Many  operators  posted  ashore  have  limited  current  experience  in  the  positions  required 
by  the  contractor’s  study 

•  There  are  few  junior  operator  positions  ashore  to  begin  with 

•  There  is  no  mechanism  to  which  a  scheduling  authority  or  a  contractor  trying  to  plan 
well  ahead  could  refer  to  identify  operators  on  a  coast-wide  basis  that  may  be  available 
for  support  at  any  given  time,  resulting  in  the  requirement  to  contact  numerous  units 
for  each  support  request 

•  There  are  often  multiple  concurrent  demands  from  contractors  or  other  sources  for  the 
limited  number  of  available  SMEs 

•  The  requirement  to  secure  appropriate  SME  support  results  in  long  lead  times  for 
contractors,  which  can  negatively  impact  work  time  lines 

•  The  current  process  of  securing  SMEs  makes  it  difficult  to  accommodate  schedule 
changes  that  come  about  due  to  extraneous  factors 

•  Contractors  often  require  a  degree  of  continuity  in  their  SMEs  for  a  period  of  days;  the 
more  important  is  the  continuity  issue  or  duration,  the  more  difficult  is  the  scheduling 

•  Scheduling  authorities  themselves  have  limited  time  and  varying  degrees  of  commitment 
to  supporting  contractors,  which  can  lead  to  less-than-ideal  assignment  of  SMEs 

5.2.2  The  Need  For  SMEs 

There  are  a  variety  of  instances  in  which  a  contractor  would  require  the  services  of  SMEs.  They 
may  be  required  to  participate  in  practical  studies  in  trainers  or  at  sea,  to  explain  particular 
processes  such  as  the  picture  compilation  process  or  the  Detect-to-Engage  sequence,  as 
interviewees  or  focus  group  members  for  Cognitive  Task  Analyses  or  team  function  analyses,  to 
provide  estimates  of  task  outcomes  for  modelling  applications,  or  as  advisors  during  any  part  of  the 
design  cycle  to  name  but  a  few.  Different  applications  imply  different  needs  in  terms  of  duration, 
number  of  cycles  for  an  individual  SME  or  groups  of  SMEs,  number  of  similar  SMEs  for  an 
individual  cycle  etc. 

5.3  Potential  Approaches 

Accepting  that  SME  support  to  contractors  will  continue  to  be  a  requirement,  it  could  be  helpful  to 
consider  the  various  means  by  which  this  could  be  accomplished.  Options  include: 

•  Current  structure 

•  Centralized  SME  availability  pool 

•  Serving  SMEs  attached  to  a  project 

•  Serving  SMEs  paid  in  their  off  time  as  consultants 

•  Independent  SME  pool  of  Navy  retirees 

•  Hybrid  approach 

5.3.1  Current  Structure 

Usually  the  contractor  identifies  the  requirement  for  SME  support  and  advises  the  Scientific 
Authority  (SA).  The  SA  will  contact  the  Naval  Liaison  Officer  (NLO)  at  the  appropriate  DRDC 


Hum'&nsystems  Incorporated 


COMDAT:  MOP  and  ORTT 


Page  67 


establishment  and  request  that  the  NLO  make  arrangements  for  the  SMEs  identified  by  the 
contractor.  Depending  on  the  NLO’s  particular  background  he/she  will  contact  a  variety  of  sources 
to  request  the  SMEs.  These  sources  might  be  Formation  personnel  pools,  operational  ships, 
training  establishments  or  a  variety  of  other  sources.  The  level  of  effort  increases  significantly  as 
the  number  of  SMEs  or  the  duration  of  the  commitment  increases. 

Advantages  of  the  Current  Structure: 

•  The  supplied  SMEs  will  normally  incur  no  additional  costs  to  the  Navy  or  the  project 

•  The  SMEs  that  meet  the  qualifications  defined  by  the  contractor  are  representational  of 
their  peer  group  and  are  normally  very  helpful 

•  SMEs  generally  already  have  the  appropriate  security  clearance  when  one  is  required 
Disadvantages  of  the  Current  Structure: 

•  A  long  lead  time  is  required  to  identify  available  SMEs 

•  This  long  lead  time  leaves  little  or  no  flexibility  to  deal  with  schedule  changes  arising 
for  unrelated  reasons 

•  The  NLO’s  task  can  be  very  challenging  and  time  consuming 

•  The  supplied  SMEs  often  do  not  meet  the  qualifications  defined  by  the  contractor  for  a 
variety  of  reasons 

•  The  supplied  SMEs  will  almost  always  be  unfamiliar  with  the  project  so  will  need  to 
be  initiated,  which  results  in  either  increased  availability  requirements  or  reduced 
employment  availability 

•  Supplied  SMEs  are  taken  away  from  pressing  duties 

5.3.2  Centralized  SME  Availability  Pool 

This  pool  would  exist  as  a  list  of  all  available  SMEs  in  a  particular  geographical  area  by 
qualification  and  availability  dates  at  least,  and  perhaps  other  criteria.  A  NLO  or  other  scheduling 
authority  need  only  consult  the  pool  and  contact  the  units  of  identified  SMEs  to  confirm 
availability. 

Advantages  of  a  Centralized  SME  Availability  Pool: 

•  Near  instantaneous  confirmation  or  disconfirmation  of  available  qualified  SMEs  for 
the  requested  period 

•  Ease  of  information  sourcing  from  the  NLO  or  scheduling  authority  perspective  would 
add  a  degree  of  flexibility  to  the  project  schedule 

•  SMEs  generally  already  have  the  appropriate  security  clearance  when  one  is  required 

•  The  supplied  SMEs  will  normally  incur  no  additional  costs  to  the  Navy  or  the  project 
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Disadvantages  of  a  Centralized  SME  Availability  Pool: 

•  A  significant  additional  administrative  workload  on  the  unit  to  identify  and  report 
personnel  availability  in  detail.  This  would  probably  result  in  reluctance  to  participate 
and  might  require  an  order,  perhaps  from  even  the  Formation  Commander. 

•  While  the  list  may  be  accurate  in  the  short  term,  it  would  get  increasingly  inaccurate 
the  farther  in  advance  one  was  planning 

•  Knowing  that  personnel  identified  on  the  list  would  be  vulnerable  to  be  called  into 
service  and  thus  lost  to  the  unit,  the  unit  may  tend  to  send  personnel  of  less  capability 
to  the  pool,  thus  making  it  non-representational  of  the  Navy  at  large. 

5.3.3  Serving  SMEs  Attached  to  a  Project 

This  approach  would  involve  assigning  serving  naval  personnel  as  a  resource  to  a  project  once  they 
have  been  identified  as  being  qualified  and  available.  These  personnel  would  remain  with  their 
unit  but  be  made  available  for  hours  or  at  most  a  few  days  at  a  time  when  required  to  support  the 
project.  If  they  were  posted,  then  either  the  responsibility  would  continue,  or  they  could  be  made 
responsible  for  identifying  and  briefing  a  successor. 

Advantages  of  Having  Serving  SMEs  Attached  to  a  Project: 

•  The  process  of  identifying  available,  suitably  qualified  SMEs  need  only  be  undertaken 
once  for  projects  with  multiple  needs  for  the  same  SME  skill  sets 

•  The  SMEs  would  become  familiar  with  the  contractor  and  the  requirements  of  the 
project,  resulting  in  reduced  orientation  requirements  and  increased  time  efficiency 

•  The  communication  channels  between  contractors  and  SMEs  would  be  significantly 
simplified,  easily  facilitating  frequent,  brief  interaction  when  required 

•  The  SMEs  would  acquire  an  understanding  of  work  being  done  outside  the  purview  of 
their  immediate  duties 

•  SMEs  generally  already  have  the  appropriate  security  clearance  when  one  is  required 

•  The  supplied  SMEs  will  normally  incur  no  additional  costs  to  the  Navy  or  the  project 
Disadvantages  of  Having  Serving  SMEs  Attached  to  a  Project: 

•  It  may  be  difficult  initially  to  identify  suitable  SMEs  that  would  be  available  for  the 
duration  of  the  project 

•  The  cumulative  time  demands  on  the  SMEs  may  conflict  with  their  primary  duties, 
meaning  one  or  the  other  has  to  give 

•  Posting  cycles  or  deployment  schedules  may  interfere  with  availability  of  assigned 
SMEs" 

•  This  approach  would  be  unsuitable  for  projects  that  by  their  nature  require  short-term 
support  from  a  wide  variety  of  different  SMEs  or  for  many  SMEs  from  one  category. 

5.3.4  Serving  SMEs  Paid  in  their  Off  Time  as  Consultants 

This  approach  would  involve  serving  SMEs  being  paid  through  the  project  for  work  they  do  on 
their  own  time,  whether  that  be  evenings,  weekends  or  statutory  holidays,  or  annual  or  unpaid  leave 
periods.  They  would  be  paid  on  an  hourly  basis  at  some  agreed  rate. 
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Advantages  of  Using  Serving  SMEs  Paid  in  their  Off  Time  as  Consultants: 

•  Neither  SMEs  nor  their  units  would  loose  productivity  during  normal  working  hours 

•  SMEs  would  be  more  available  because  the  consultant  would  not  be  competing  for 
scarce  availability  during  normal  working  hours 

•  NLOs  or  scheduling  authorities  need  only  identify  potential  candidates;  it  would  be  up 
to  the  contractor  and  the  SMEs  to  come  to  terms  on  arrangements  on  a  non¬ 
interference  basis  with  the  Navy 

•  The  contractor  would  be  more  likely  to  get  the  SMEs  best  suited  to  the  requirements  of 
the  project 

•  SMEs  generally  already  have  the  appropriate  security  clearance  when  one  is  required 
Disadvantages  of  Using  Serving  SMEs  Paid  in  their  Off  Time  as  Consultants: 

•  Conflicts  may  arise  when  the  SME  support  must  be  during  normal  working  hours;  the 
SME  would  have  to  ask  for  leave  or  time  off  from  the  unit 

•  Posting  cycles  or  deployment  schedules  may  interfere  with  the  availability  of  identified 
SMEs" 

•  The  degree  to  which  this  idea  is  contrary  to  existing  DND  policies  is  unknown 

•  DND  is  essentially  paying  for  their  own  expertise 

•  This  approach  could  be  perceived  by  the  Navy  as  facilitating  the  recruitment  of 
experienced  personnel  away  from  the  Navy  into  a  contractor’s  organization 

5.3.5  Independent  SME  Pool  of  Navy  Retirees 

This  pool  of  experts  would  be  called  upon  as  paid  consultants  on  an  as-required  basis.  They  may 
be  fully  retired  or  engaged  in  a  second  career,  and  would  generally  work  on  an  ad-hoc  basis.  The 
size  and  composition  of  the  pool  would  vary  with  time  depending  on  projected  requirements  and 
the  success  of  attracting  suitable  candidates  to  the  pool. 

Advantages  of  an  Independent  Pool  of  Navy  Retirees: 

•  The  effort  required  to  establish  and  maintain  the  pool  would  be  totally  independent  of 
Navy  resources 

•  Contractors  would  have  the  freedom  to  select  only  suitable  SMEs  into  the  pool,  and 
then  to  select  which  ones  best  meet  the  requirements  for  any  given  project 

•  A  well  established  pool  would  provide  the  contractor  with  the  flexibility  required  to 
react  to  short  notice  schedule  changes  or  opportunities  to  access  high  demand 
resources  such  as  naval  trainers 

•  The  pool  could  readily  be  used  for  support  during  stages  of  a  project  when  the 
participation  of  SMEs  with  extremely  current  or  very  specific  experience  is  not  critical 

Disadvantages  of  an  Independent  Pool  of  Navy:  Retirees: 

•  It  could  take  considerable  effort  to  find  SMEs  with  the  required  qualifications, 
experience  and  currency,  and  who  are  available  for  part-time,  ad-hoc  work 

•  Security  clearances  for  the  pool  members  would  become  the  responsibility  of  the 
maintaining  agency 
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•  The  overall  maintenance  of  the  pool  would  represent  a  cost  to  the  maintaining  agency 
that  would  have  to  borne  either  by  that  agency  or  by  DND 

•  There  would  have  to  be  regular  scheduled  turnover  within  the  pool  as  retirees  skill  sets 
and  familiarity  with  up  to  date  practises  would  deteriorate  over  time 

•  There  would  be  increased  administration  to  have  the  list  of  potential  SMEs  for  a  given 
project  added  to  the  bid  or  contract 

•  There  would  be  a  cost  to  DND  for  the  provision  of  the  SME  services 

5.3.6  Hybrid  Approach 

There  are  significant  advantages  and  disadvantages  to  all  of  the  approaches  discussed  above.  There 
may  in  fact  be  no  requirement  to  choose  one  over  another;  with  the  concurrence  of  the  appropriate 
authorities  an  approach  could  be  adopted  that  leverages  the  advantages  of  the  various  options  while 
minimizing  the  disadvantages.  In  this  way,  a  variety  of  mechanisms  for  acquiring  the  appropriate 
SME  support  for  any  given  project  could  be  used  depending  on  the  specific  natures  of  the  project 
and  of  the  SME  support  required. 

Advantages  of  a  Hybrid  Approach: 

•  The  majority  of  support  required  for  a  project  would  be  identified  up  front,  resulting  in 
an  understanding  by  all  stakeholders  of  the  scope  of  the  support 

•  The  various  components  of  the  support  required  could  readily  be  associated  with  the 
most  effective  means  of  providing  that  support 

•  Trade  offs  can  be  made  among  stakeholders  to  arrive  at  a  compromise  between 
administrative  requirements  to  arrange  support  on  a  case  by  case  basis,  the  requirement 
to  pull  operational  personnel  from  their  regular  duties,  and  cost  to  DND  to  provide 
contracted  support 

•  The  support  ultimately  provided  to  the  contractor  would  be  more  suited  to  the 
requirements  with  a  reduction  on  the  overall  administrative  load  of  the  Navy 

Disadvantages  of  a  Hybrid  Approach: 

•  Because  several  of  the  available  options  involve  contracting  SMEs,  there  would  be  a 
cost  to  DND 

•  Policies  to  facilitate  some  of  the  options  would  have  to  be  researched  to  ensure  they 
exist,  or,  if  they  do  not,  policies  would  have  to  be  drawn  up  and  approved 


5.4  Recommended  Approach 

While  there  is  no  obvious  single  best  option,  to  effectively  support  contractor  requirements,  we 
recommend  a  hybrid  approach  consisting  of  elements  of  the  current  system  (2.1),  attaching  serving 
SMEs  to  a  project  (2.3),  and  paying  serving  SMEs  in  their  off  time  as  consultants  (2.4). 

The  idea  of  a  centralized  SME  pool  (2.2)  was  ruled  out  because  the  administrative  demands  on  the 
Navy  to  effectively  manage  the  pool  would  outweigh  the  benefits  of  the  ready  availability  of 
SMEs.  Likewise,  the  idea  of  an  independent  SME  pool  of  retirees  (2.5)  was  ruled  out  because  of 
the  excessive  effort  required  on  the  part  of  the  contractor  to  build  and  maintain  the  pool.  The 
overhead  of  recruiting  and  administration  for  people  infrequently  called  upon  would  have  to  be 
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passed  on  to  DND  in  order  to  make  this  a  viable  option  for  the  contractor,  likely  resulting  in 
inordinately  high  costs  to  DND. 

For  contractors  to  fulfil  the  mandate  given  them  with  respect  to  SME  input  for  a  DND  contract,  a 
balance  has  to  be  reached  among  three  competing  components.  These  components  are: 

•  first,  the  SME  input  required  to  provide  valid  and  accurate  information  for  the  study, 

•  second,  the  demands  on  Navy  resources  to  arrange  for  and  deliver  suitable  SMEs  and 

•  third,  the  end  cost  to  DND  for  the  support. 

The  recommended  approach  would  build  on  the  current  system  of  providing  support  as  opposed  to 
replacing  it.  Recognizing  that  each  project  will  have  its  own  unique  support  requirements,  having 
the  flexibility  to  arrange  for  one  or  more  particularly  valuable  SMEs  to  be  attached  as  support  to 
the  project  on  an  as-needed  basis,  or  to  pay  serving  SMEs  on  a  short  term,  ad  hoc  basis  for  specific 
support  requirements  would  go  a  long  way  to  effectively  meeting  the  balance  among  the  competing 
requirements.  Although  the  approach  of  paying  serving  SMEs  in  their  off  time  would  admittedly 
add  expense  to  the  project,  the  benefits  of  ensuring  the  right  support  is  available  at  the  critical  time 
outweigh  the  costs  when  the  overall  risk  to  the  project  is  considered. 

The  major  challenges  to  achieving  this  would  include: 

For  attaching  serving  personnel  to  a  project: 

•  For  this  to  be  successful  either  individual  unit  commanders  would  have  to  agree  to  allow 
personnel  under  their  command  to  participate,  or  a  higher  authority  (CMS,  Formation 
Commander  or  MAROPSGRU  Commander)  would  have  to  put  in  place  a  policy 
whereby  units  commanders  were  required  to  cooperate  to  the  extent  possible  given 
current  demands  on  their  personnel.  The  former  would  likely  require  a  significant 
investment  of  effort  on  the  part  of  an  NLO  to  negotiate  with  commanders  on  a  case-by¬ 
case  basis.  The  latter  would  likely  take  a  significant  one  time  investment  of  time  by 
DRDC  to  convince  a  higher  authority  of  the  value  of  this  approach 

For  paying  serving  SMEs  in  their  off  time  as  consultants: 

•  If  there  was  a  large  volume  of  work  assigned  to  individuals  and  paid  by  contractors, 
suitable  SMEs  might  have  to  invest  the  time  and  effort  to  set  themselves  up  as  self- 
employed  consultants.  This  could  include  registering  themselves  as  a  business  and 
maintaining  all  required  paperwork,  including  GST  management.  We  believe  the 
annual  threshold  is  $3 Ok,  and  that  this  is  unlikely  to  be  reached  in  the  great  majority  of 
cases.  An  alternative  might  be  to  make  payments  through  navy  pay  channels. 

•  Currently  there  appear  to  be  barriers  within  DND  to  having  SMEs  work  as  paid 
consultants  on  DND  business.  While  any  cultural  barriers  could  be  overcome  with 
time,  any  administrative/policy  barriers  impeding  the  lawful  employment  of  currently 
serving  SMEs  would  have  to  be  addressed  up  front.  This  would  likely  take  a 
significant  one-time  investment  of  time  by  DRDC  to  achieve. 
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6.  Overall  Summary 


This  body  of  work  analyses  the  suitability  and  practicality  of  using  the  ORTT  Navy  training 
records  as  an  alternative  to  NCOT  for  the  collection  of  COMDAT  TD  relevant  MOPs.  An  analysis 
methodology  was  developed  for  the  extraction  of  MOP  data,  and  a  range  of  suitable  MOPs  was 
demonstrated  as  being  practically  accessible  from  the  data  record.  It  was  concluded  that  the  ORTT 
scenario  records  would  be  more  suitable  to  collect  MOP  data  than  conducting  a  dedicated  T&E 
trial  in  NCOT.  Finally,  a  recommended  approach  was  described  by  which  the  Navy  can  provide 
SMEs  to  support  future  R&D  efforts,  without  incurring  an  unacceptable  administrative  and 
manpower  burden. 
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7.  List  of  Acronyms 


AAWC 

Anti  Air  Warfare  Commander 

ALG 

Algonquin 

AOP 

Area  of  Probability 

ARRO 

Air  Raid  Reporting  Operator  (formerly  RT1) 

AS 

Anti  Surface  Warfare  Commander  (Also  ASuWC) 

ASPO 

Anti-submarine  Plotting  Operator  (formerly  RT2) 

ASuWC 

Anti  Surface  Warfare  Commander  (Also  AS) 

ASWC 

Anti  Submarine  Warfare  Commander 

ASWD 

Anti  Submarine  Warfare  Director 

AX 

Anti  Submarine  Warfare  Commander 

CAP 

Combat  Air  Patrol 

CO 

Commanding  Officer 

CCS 

Command  and  Control  System 

CSTC 

Combat  Systems  Training  Centre 

CTA 

Cognitive  Task  Analysis 

CTG 

Commander  Task  Group 

C&C 

Command  and  Control 

C2 

Command  and  Control 

ESM 

Electronic  Support  Measure 

EWS 

Electronic  Warfare  Supervisor 

FCS 

Fire  Control  Supervisor 

HAL 

Halifax 

Helo 

Helicopter 

MOP(s) 

Measure(s)  of  Performance 

MPA 

Maritime  Patrol  Aircraft 

MSDF 

Multi  Sensor  Data  Fusion 

NCOT 

Naval  Combat  Operator  Trainer 

OOW 

Officer  of  the  Watch 

ORO 

Operations  Room  Officer 

ORTT 

Operations  Room  Team  Trainer 

POC 

Proof  of  Concept 

PU 

Participating  Unit 

REG 

Regina 

ROE 

Rules  of  Engagement 

SAC 

Ship-borne  Air  Controller 

SAG 

Surface  Action  Group 

SCS 

Sonar  Control  Supervisor 

SITREP 

Situation  Report 

SME 

Subject  Matter  Expert 

SWC 

Sensor  Weapons  Controller 

TG 

Task  Group 

T&E 

Test  &  Evaluation 

VAN 

Vancouver 
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Annex  A:  Ability  of  the  ORTT  to  support 
T&E  MOPs 


Specific  problems  in  collecting/conducting  MOPs  are  emphasised  in  bold  text.  Numbers  refer  to 
MOP  labels  in  prior  reports 


DETAILED  MEASURES 

FEASIBILITY  IN  ORTT/  COMMENTS 

T&E  TASK  1  Check  mission  picture  2.1.1 

la  Accuracy  in  detecting  relevant  info  within  incoming 
message  stream 

Can  get  information  from  audio  playback  and  real-time  scenario 
monitoring 

Messages  passed  by  hand  (non-scripted)  can't  get  from  digital 
record  obtainable  through  real-time  monitoring  (make  note  of 
content  and  when  passed)  and  video  record  for  time  of  occurrence. 
Amount  of  paper  messages  here  is  1%  of  real-life 

Can  send  text  messages  to  signalman  through  CCS,  but  not  used 
often 

1b  Accuracy  in  ignoring  irrelevant  info 

N/A  in  ORTT  b/c  no  irrelevant  info  inputted 

1c  Total  time  spent  in  comms  dealing  with  incoming  info 

Could  get  from  audio  track  and  video;  labour  intensive 

Id  Time  to  detect  high  salience  message 

Voice  message  -  observe  acknowledgement  or  operator  action  by 
way  of  video  playback  or  real-time  monitoring 

1e  Accuracy  in  requesting  additional  info  (i.e.  number  of 
messages  that  required  follow  up  that  resulted  in  ORO 
request  for  more  info) 

Possibly- we  don't  have  the  control  needed  (may  do  ad-hoc) 

If  Accuracy  in  ignoring  lower  priority  messages 

No  -  not  done  in  training  scenario  (only  at  sea)  -  may  happen  by 
chance 

T&E  TASK  2.  Relate  new  info  Ops  Room  picture  2.1.2 

2a  Accuracy  in  directing  (communicates)  info/direction 
resulting  from  message 

Can  get  from  video,  audio,  rely  on  SME  to  determine  if  correct 

2b  Accuracy  re  message  content  in  briefing  appropriate  stn 

Same  as  2a. 

2c  Accuracy  in  comprehension  of  impact  on  pre¬ 
plans/response  options/tactical  situation 

Can  do  in  ORTT ;  team  must  be  put  under  conflict  then  ORO  must 
make  decision 

2d  Accuracy  in  recognition  of  impact  on  Ops  Room  capability 

Can  simulate  this  in  ORTT  2  if  the  team  is  getting  good,  game 
controllers  may  input  equipment  malfunctions,  etc.  but  not  likely 

2e  Total  time  spent  in  relaying  info  re  Ops  Room  status 

T&E  TASK  3  Relate  (new  info)  to  RMP  Includes  aspects  of  build  awareness  across  domains  (RAP,  MSP,  MsubP). 

3a  Accuracy  of  salient  info  within  RMP  prior  to  new  message 

N/A  b/c  can’t  do  probes;  can  only  infer  knowledge  of  ORO  from 
comms,  may  get  through  team  briefing?  (high  level  briefings,  not 
done  regularly).  Maybe  we  can  use  tags? 

3b  Accuracy  of  salient  info  within  RMP  after  new  message 

same  as  3a 

3c  Time  for  ORO  to  complete  understanding  of  new  info 
regarding  RMP 

same  as  3a 

3d  Accuracy  in  taking  appropriate  action  as  a  result  of 
updated  RMP 

same  as  3a. 

Ample  salient  info  presented  to  ORO. 

Master  Events  List  will  be  helpful. 

CFNOS  testing  criteria/scoring  sheets  &  would  it  be  helpful 

3e  Total  time  spent  on  SSD  relating  new  info  to  RMP 

May  be  complex  to  assemble. 

3f  Accuracy  in  assessing  the  integrated  tactical  picture 

Can  get  from  multiple  threats  and  threat  priorities;  can't  control  the 
scenario  but  can  infer  from  SME  assessment  of  scenario  playback 
(comms  and  video) 

Human, systems  Incorporated 


COMDAT:  MOP  and  ORTT 


Page  A-l 


DETAILED  MEASURES 

FEASIBILITY  IN  ORTT/  COMMENTS 

T&E  TASK  4  Relate  new  info  to  RAP  2.1.5.  (as  an  example):  Ensure  contact  classified  2.4.3. 1 

Note:  Same  approach/measures  to  be  used  in  principle  for  MSP  and  MsubP 

SUB  TASK  4.1  Identify  friendly  aircraft  Few,  if  any  occurrences  in  ORTT  scenario. 

4.1a  Accuracy  in  identifying  friendly  aircraft  (by  type/mission) 

Available  from  scenario  ground  truth.  May  assess  at  RT 1  level. 

4.1b  Mean  time  spent  in  identifying  friendly  aircraft 

See  4.1a 

4.1c  Total  number  of  queries  or  total  time  spent  in  querying 
team  for  additional  info 

See  4.1a 

SUB  TASK  4.2  Identify  hostile/suspect  aircraft 

Hostile  /  suspect  a/c  will  be  occasional  insertions  in  main  air  contact  data  stream. 

4.2a  Accuracy  in  identifying  hostile  /  suspect  aircraft  (by 
aircraft  type) 

Can  do  if  we  have  ground  truth 

4.2b  Mean  time  spent  in  identifying  hostile/suspect  aircraft 

Can  get;  beginning  time  is  when  track  first  initiated  by  ATAI  (auto 
track,  auto  initiate)  if  we  know  ground  truth 

4.2c  Total  number  of  queries  or  total  time  spent  in  querying 
team  for  additional  info 

Can  get  from  playback 

SUB  TASK  4.3  Identify  neutral  aircraft 

4.3a  Accuracy  in  identifying  neutral  aircraft 

Same  as  4.2;  neutral  =  “non-threat”,  military  unit;  if  not  put  in 
game,  can  probably  ask  for  one 

4.3b  Total  time  spent  in  identifying  neutral  aircraft 

Same  as  4.2 

4.3c  Total  number  of  queries  or  total  time  spent  in  querying 
team  for  additional  info 

Same  as  4.2 

SUB  TASK  4.4  Identify  NU  tracks  (non-updated) 

4.4a  Accuracy  in  identifying  NU  tracks 

Not  normally  included  in  scenario  in  ORTT 

(NU  track  is  when  track  quality  deteriorates  to  1 ;  when  it  starts 
flashing  =  NU.  Normally  associated  with  link) 

4.4b  Total  time  spent  in  identifying  NU  tracks 

Same  as  4.4a 

4.4c  Total  number  of  queries  or  total  time  spent  in  querying 
team  for  additional  info 

Same  as  4.4a 

SUB  TASK  4.5  Identify  tracks  reported  by  ownship  or,  conversely,  by  other  participating  units  (Pus) 

4.5a  Accuracy  identifying  air  tracks  being  reported  by  ownship 

Not  a  relevant  measure 

4.5b  Total  time  spent  locating  air  tracks  being  reported  by 
ownship 

Not  a  relevant  measure 

4.5c  Total  number  of  queries  or  total  time  spent  in  querying 
team  for  additional  info 

Not  a  relevant  measure 

SUB  TASK  4.6  Identify  own  force  engagement  status  This  measure  will  only  be  used  if  scenario  requires  engagement 

4.6a  Accuracy  identifying  air  tracks  being  engaged  by  ownship 
or  ships  in  TG 

Needs  a  T&E  probe-probably  not  available 

4.6b  Total  time  spent  in  locating  air  tracks  being  engaged 

Should  be:  “response  time  to  be  ready  to  engage  correct  track”  b/c 
may  purposely  decide  to  wait  to  engage. 

This  could  be  a  measure  for  SWC  or  ORO 

4.6c  Total  number  of  queries  or  total  time  spent  in  querying 
team  for  additional  info 

Should  be:  “total  time  querying  team  for  info  (not  “additional  info”) 
Can  get  from  comm  playback 

SUB  TASK  4.7  React  to  threat  track  symbology  LINKed 

1  by  consort  not  on  video 

4.7a  Time  to  recognise  symbology  not  on  video 

HI  A  in  ORTT 

4.7b  Time  to  order  remedial  action 

HI  A  in  ORTT 

SUB  TASK  4.8  React  to  LINK  not  gridlocked 

4.8a  Time  to  recognise  LINK  not  gridlocked 

HI  A  in  ORTT 

4.8b  Time  to  order  remedial  action 

HI  A  in  ORTT 
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DETAILED  MEASURES 

FEASIBILITY  IN  ORTT/  COMMENTS 

T&E  TASK  7  Assess  threats  -generic 

SUB  TASK  7.1  Detect  changes  in  tactical  situation 

7.1a  Accuracy  in  ID  new  threats 

Can  easily  get  accuracy  and  ID.  Lots  of  threats  in  scenario. 

7.1b  Accuracy  in  ID  changes  in  threat  status 

Lots  of  opportunity  provided  in  scenarios  for  misidentifications 

7.1c  Time  to  detect  new  threat 

Should  be:  “time  to  ID  new  threat”. 

Easily  achieved  from  playback  if  flag  assigned  by  T&E  team  during 
scenario  execution 

Not  ORO  measure;  more  a  team  or  even  task  group  measure. 

7.1  d  Time  to  detect  change  in  threat  status 

Same  as  7.1c 

7.1  e  Time  to  provide  SITREP 

Maybe  ad-hoc  but  difficult  to  plan/guarantee  in  ORTT 

May  be  able  to  ask  trainer  to  ask  for  ORO  sitrep 

7. If.  Accuracy  in  SITREP  contents 

same  as  7.1  e;  decompose  after  and  get  ground  truth 

SUB  TASK  7.2  Determine  threat  priorities  across  domains 

Will  need  to  vary  circumstances  to  ensure  that  some  threat  changes  occur  in  a  particular  domain  while  the 
focus  is  in  that  domain,  at  other  times  when  focus  is  in  another  domain 

7.2a  Accuracy  in  ranking  threat  priorities  across  domains 

Likely  not  feasible  in  ORTT 

We  know  ground  truth  but  don't  know  what  team  knows;  need  to 
infer  accuracy  based  on  what  team  knows.  May  use  SME 
assessment  of  team  behaviour. 

7.2b  Time  to  assess  threat  priorities  across  domains 

May  be  feasible  from  determining  onset  time  of  threats  and  orders 
to  team. 

SUB  TASK  7.3  Assess  threats-  (air  warfare  specific)  Identify  air  track  threat  priority 

Similar  MOPs  should  apply  in  principle  to  other  warfare  areas. 

7.3a  Accuracy  in  identifying  threat  priorities 

Done  by  RT 1  as  part  of  resolve  procedure. 

7.3b  Total  time  to  identify  three  highest  priority  threats 

N/A;  perhaps  time  to  resolve  is  better  measure? 

7.3c  Time  to  annotate  CCS  with  CPA 

N/A;  perhaps  time  to  resolve  is  better  measure? 

Should  be  able  to  identify  when  it  is  resolved  (i.e.  endpoint); 
maybe  when  resolved  on  SSD  (identifying  track). 

7.3d  Accuracy  in  determining  CPA 

N/A;  perhaps  time  to  resolve  is  better  measure? 

7.3e  Accuracy  in  determining  lethality 

Not  relevant;  related  to  threat  priority 

7.3f  Time  to  determine  lethality 

Same  as  7.3e 

7.3g  Time  to  create  a  SITREP 

Cannot  insert  T&E  requirement  for  SITREP:  may  need  to  rely 
on  when  CO  asks  for  one 

7.3h  Accuracy  and  completeness  of  SITREP 

As  for  7.3g 

SUB  TASK  7.4  Analyse  history  profile13  of  hostile  aircraft 

7.4a  Accuracy  in  analysing  attack  history  of  hostile  aircraft 

Not  relevant;  no  repeated  action  in  scenarios 

7.4b  Time  to  analyse  attack  history  of  hostile  aircraft 

Not  relevant;  no  repeated  action  in  scenarios 

7.4c  Accuracy  in  assessing  number  of  weapons  remaining  on 
hostile  aircraft 

Determine  from  ORO/SWC  comms 

7.4d  Total  number  of  queries  or  total  time  spent  in  querying 
team  for  additional  info 

Same  as  7.4c 

7.4e  Accuracy  in  assessing  attack  tactics  used  by  hostile 
aircraft  (individual  contacts) 

Not  relevant 

13 

By  history  profile  we  mean  the  patterns  of  trajectory  shown  over  time  by  a  particular  contact  that  is  potentially  hostile;  these 
patterns  include  changes  in  altitude,  speed  and  direction,  communication  trends,  EW  emissions  etc. 
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DETAILED  MEASURES 

FEASIBILITY  IN  ORTT/  COMMENTS 

T&E  TASK  9  Assess  sensors:  Sensor  information  can  become  degraded  through  equipment  malfunctions  or  restrictions.  Hence, 
will  need  to  ensure  that  sensor  becomes  degraded  during  scenario 

9a  Accuracy  assessing  that  sensors  are  less  than  optimum 

Not  feasible  in  existing  scenarios 

9b  Time  to  appreciate  sensor  performing  less  than  optimum 

Not  feasible  in  existing  scenarios 

9c  Accuracy  in  identifying  current  sensor  range  predictions 

Not  feasible  in  existing  scenarios 

9d  Total  time  in  identifying  current  sensor  range  predictions 

Not  feasible  in  existing  scenarios 

T&E  TASK  10  Manage  ship  surveillance 

10a  Accuracy  recognising  problems  in  detect  to  resolve 
process 

SME  reviews  ORO  actions  in  replay 

10b  Time  to  recognise  problems  in  detect-to-resolve  process 

Can  determine  from  micro  analysis  of  replay 

10c  Appropriateness  of  remedial  action  to  correct  errors 

SME  reviews  ORO  actions  in  replay 

Table  A1 :  Summary  of  feasibility  of  collecting  MOP  data  in  the  ORTT 
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Annex  B:  Feasibility  and  methods  for  providing 

information  to  ORO  and  for  capturing  MOPs 


Means  of 
providing  info 
to  ORO 

(other  than  T&E 
software) 

Feasible 

Timing  Sent 

Timing 

Reed 

T&E  - 

who  does? 

If  not  T&E, 
ID  source 

Capture 

Response 

Action 

Capture 

Response 

Content 

ID  target  of 
response 

Time  of 
response 

Assess 

Response 

Text  message 
via  CCS 

NO 

Text  message 
by  hand 

YES:  real-time 

observation  or 
video  record 

YES:  need  to 

add  flag  to 
T&E  video 
record 

YES:  obs 
notes 

ORO  obs,  trial 
coord  from 
video  record 

YES:  real-time 
observation  or 
scenario 
playback 

YES:  real-time 
observation  or 
scenario 
playback 

YES:  real¬ 
time 

observation 
or  scenario 
playback 

YES:  from 
scenario 
playback 
clock  or  obs 
record 

Post  event 
analysis  by 
SME 

Voice  message 
from  net 

YES 

playback  of 
comm  record 

YES 

Playback 

clock: 

YES: 

playback 

clock 

Not  required 

YES: 

video/audio 

record 

YES:  ORTT 
video/audio 
record  of  ORO 

YES:  ORTT 

video/audio 
record  of  ORO 

YES:  ORTT 

video/audio 
record  of 

ORO 

YES:  from 
scenario 
playback 
clock  or  obs 
record 

Post  event 
analysis  by 
SME 

Voice  message 
face  to  face 

YES:  from 

T&E 

video/audio 
record  or  real¬ 
time  obs 

YES:  need  to 
add  clock  to 
T&E  video 
record 

YES: 

Not  required 

YES: 

video/audio 

record 

YES:  ORTT 
video/audio 
record  of  ORO 
or  scenario 
playback 

YES:  ORTT 
video/audio 
record  of  ORO 
or  scenario 
playback 

YES:  ORTT 
video/audio 
record  of 

ORO  or 
scenario 
playback 

YES:  from 
scenario 
playback 
clock  or  obs 
record. 

Post  event 
analysis  by 
SME 

Table  B1:  Methods  assessed  in  ORTT  for  capturing  data  for  MOPs 
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Input  events 
from  scenario 

Time  into 
scenario 

Time 

painted 

Time  into  AOI 

Time 

symbology 

Time 

Out 

Time  change 
in  status 

Time  at  own 
ship  weapons 
range 

Time  at  A/C 
weapons 
range 

Time  EWS 
emitted 

Time  EWS 
detected 

Background 

air/surface- 

commercial 

Background 

surface 

Hostile  air 

Hostile  surface 
A/C  changes 
course  to  TG 

A/C  changes 
speed 

A/C  changes  alt 
A/C  radar 

A/C  changes 
previous  pattern 
A/C  assumes 
attack  profile 
Neutral-changes 
profile 

ASCACT  patterns 

Dependent 
on  scenario 
May  be  able 
to  get  Master 
Events  List 
from 

trainers.  T&E 

personnel 

insert  flags 

during 

scenario 

execution. 

YES: 

master  events 
list  provided 
by  trainer  or 
scenario 
playback. 

T&E 

personnel 
insert  flags 
during 
scenario 
execution. 

YES:  scenario 
playback  and 
observing 
appropriate 

SSD 

T&E  personnel 
insert  flags 
during  scenario 
execution. 

YES:  from 

scenario 

playback 

T&E  personnel 
insert  flags 
during  scenario 
execution. 

YES: 

scenario 

playback 

T&E 

personnel 
insert  flags 
during 
scenario 
execution. 

YES:  master 
events  list 
provided  by 
trainer  or 
scenario 
playback 

T&E 

personnel 
insert  flags 
during 
scenario 
execution. 

YES:  scenario 
playback 

YES:  scenario 
playback 

YES:  from 
master  events 
list  provided 
by  trainer  or 
scenario 
playback 

YES: 

scenario 

playback 

ORO  actions 

Scenario  playback 

Audio/video  record 

Scenario  playback 

Time  stamped  audio/video  record 

ORO  action  on 
CCS 

YES.  Scenario  playback 

YES:  Scenario  playback 

ORO  consults  other  sources 

ORO  use  of  ancillary  information  (e.g.  manuals,  plot,  simulated  stateboards,  tacpacs)  captured  by  ancillary  video 
and  analysed  post  event.  May  also  be  captured  by  direct  ORO  observation. 

Table  B2:  Scenario  events  and  ORO  actions  assessed  for  capturing  MOP  data 
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ORO  Communications 

Capture  message  content 

Capture/  timestamp  action 

ID  recipient 

Assess  content 

ORO  speaks  directly  to 
team/member  (not  on  net) 

YES:  separate  audio/video  record 

YES.  separate  audio/video  record. 

YES.  Separate  audio/video 
record 

SME  analysis  of  audio/video 
record 

ORO  uses  net 

YES:  ORTT  audio  record 

YES:  ORTT  audio  playback  plus  CCS 
clock 

YES:  playback  of  audio 
circuit 

SME  analysis  of  T&E  audio 
playback 

Assessing  ORO 
knowledge 

Real  time 
SITREP  to  CO 
Only 

achievable  if 
CO  requests 

ORO CCS 

Display 

Bogus  request  for 
info 

Response  time  to 
embedded  probe 

Picture  content  at  start  of 
scenario 

YES:  link  audio 
file  with 
scenario 
playback  - 
depends 

YES:  scenario 
playback 

NO 

NO 

Change  in  picture  content 

YES:  scenario 
playback 

YES:  scenario 
playback 

YES:  scenario 
playback 

YES:  scenario 
playback 

Indiv  threat  comprehension 

YES:  scenario 
playback 

YES:  scenario 
playback 

YES:  scenario 
playback 

NA 

Change  in  threat  status 

YES:  scenario 
playback 

YES:  scenario 
playback 

YES:  scenario 
playback 

YES:  scenario 
playback 

Relative  threat  priorities 

YES:  scenario 
playback 

YES:  scenario 
playback 

YES:  scenario 
playback 

Tactical  situation 

YES:  scenario 
playback 

YES:  scenario 
playback 

Time  to  CPA 

YES:  scenario 
playback 

YES:  scenario 
playback 

YES:  scenario 
playback 

Own  engagement 
envelope:  earliest/latest 
point  to  shoot  - 

YES:  scenario 
playback 

YES:  scenario 
playback 

Table  B2:  Scenario  events  and  ORO  actions  assessed  for  capturing  MOP  data  (continued) 
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Annex  C:  Scenario  data  record 


C-1:  Data  record  for  selected  air,  subsurface  and  surface  tracks 


ORO 

SWC 

ARRO 

CANEWS 

ASWC 

Time 

Events 

Comments 

Send 

Receive 

Send 

Receive 

Send 

Receive 

Send 

Receive 

Send 

Receive 

9:18:40 

Air  Track  2400 
appears,  this  is  a 
new,  unassessed  air 
track 

9:18:44 

ORO  hooks 

Track2400 

9:18:54 

AWW 1  net  / 
to  team  - 
resolve  2400 

9:19:03 

No  IFF  implies  non¬ 
commercial 

AWWI: 
course, 
speed  &  neg 
IFF 

9:19:07 

AWWI:  Neg 
ESM 

9:19:09 

2400  made  suspect 
by  RT1 

Adds  to 
suspicion 

AWWI:  issue 
warning  1  to 
suspect  2400 

9:20:14 

ORO  adjusts 
rangescale  to  128 
and  hooks  Track2400 

9:21:40 

C&C  -ORO 
asks  SWC  if 
there  might  be 
any  surface 
contacts  that 

2400  might  be 
passing 
targeting  info 
to.  SWC 
apparently 
misinterprets 
the  question 
and  responds 
that  he  agrees 
that  2400  is 
probably  an 

C&C  -ORO 
asks  SWC  if 
there  might  be 
any  surface 
contacts  that 
2400  might  be 
passing 
targeting  info 
to.  SWC 
apparently 
misinterprets 
the  question 
and  responds 
that  he  agrees 
that  2400  is 
probably  an 
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ORO 

SWC 

ARRO 

CANEWS 

ASWC 

Time 

Events 

Comments 

Send 

Receive 

Send 

Receive 

Send 

Receive 

Send 

Receive 

Send 

Receive 

MPA 

MPA 

9:22:43 

ORO  informed  that 
warning  3  being 
issued  to  Track2400 

Miscommuni 

cation 

between 

ORO  &  SWC 
about  what 
2400  is 

C&C  -  SWC 
to  ORO  - 
update  that 
AAWC 
issuing 

Warning  3  to 
suspect  2400 

C&C  -  SWC 
to  ORO  - 
update  that 
AAWC 
issuing 

Warning  3  to 
suspect  2400 

9:23:04 

New  air  Track2404  on 
Link  from  REG;  RT1 
hooks  Track2404 

9:23:08 

SPS  49  radar 
generated  local  air 
track  8011  in 
commercial  air  lane 

only  RT1 
knows  it 
exists  based 
on  his  range 
scale 

9:23:16 

HAL  manually  putin 
altitude  and  speed 
after  conversation 
with  REG  on  AAW 
Coord  external 

new  air  track 
in 

commercial 
air  lane 

9:23:30 

REG  released  track  to 
link  before  HAL 
released  track  to  link, 
therefore  local  and 
remote  tracks 
superimposed; 
required  manual 
intervention  by  RT 1  to 
correlate 

9:23:30 

SWC  has  altitude  & 
new  speed  on 
Track2404;  reporting 
unit  is  REG. 

both  local  and 
link  track 
numbers  are 
showing  on 
CCS  screen 
for  this  air 
contact; 
clutters  display 

9:24:04 

RT  1  made  2404 
assumed  friend  based 
on  speed  and  air  lane 
even  though  no  IFF 

9:24:30 

SWC  informs  ORO 
that  they've  been 
directed  to  re-issue 

air 

Track2404 

made 

C&C  -  SWC 
jumps  to 

C&C  to  brief 

C&C  -  SWC 
jumps  to 

C&C  to  brief 
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ORO 

SWC 

ARRO 

CAN  EWS 

ASWC 

Time 

Events 

Comments 

Send 

Receive 

Send 

Receive 

Send 

Receive 

Send 

Receive 

Send 

Receive 

Warning  2  to 

Track2400 

assumed 
friend  even 
though  no 

IFF 

information 

ORO  that 
they've  been 
directed  to 
re-issue 
Warning  2  to 
2400 

ORO  that 
they've  been 
directed  to 
re-issue 
Warning  2  to 
2400 

9:27:08 

EWS  told  SWC  that 
air  Track2400  is 
Mainstay  aircraft 
based  on  EWS 
information 

AWW  1  - 
EWS  to  SWC 
reporting  rkt 
2217  and 
correlated 
with  2400 
and 

concluded 
carried  on 
Mainstay 
aircraft  (so 
know  type  of 
aircraft) 

AWW  i  - 
EWS  to  SWC 
reporting  rkt 
2217  and 
correlated 
with  2400 
and 

concluded 
carried  on 
Mainstay 
aircraft  (so 
know  type  of 
aircraft) 

9:28:04 

ORO  hooked  air 

Track  2400 

9:29:08 

REG  reporting 
Track2400  is  hostile; 
ORO  on  HAL 
recommends 

Track2400  be  made 
hostile 

C&C - ORO 
recommended 
air  Track  2400 
(assessed 
Mainstay)  be 
made  hostile 

9:35:20 

AAW  Coord  external  - 
HAL  to  AAWC  - 
recommend  have  2 
friendlies  (suspect 

CAP)  and  sending 
them  over  to  kill 
Track2400 

ORO  tells 
team  to 
make  air 
Track2400 
hostile  based 
on  REG 
reporting 
Track2400  is 
hostile 
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ORO 

SWC 

ARRO 

CANEWS 

ASWC 

Time 

Events 

Comments 

Send 

Receive 

Send 

Receive 

Send 

Receive 

Send 

Receive 

Send 

Receive 

9:59:55 

HAL  sonar  (HMS) 
picked  up  sub  surface 
contact3  NM  from 

HAL 

HAL 

recommendi 
ng  to  Anti-air 
Warfare 
Commander 
of  Task 

Group  to 
have  2 
friendly 
aircraft  kill 
hostile 
Track2400 

ASW 

External 
(Action  A  net) 

-  HAL  ASWC 
reported 
"hot'Vsonar 
contact3  NM 
from  HAL 

10:00:05 

HAL  conducted 
urgent  torpedo  attack 
on  new  sonar  contact, 
Poss  sub  2275 

HAL  hull- 
mounted 
sonar  pickup 
sub-surface 
contact  3 
nautical 
miles  away 

ASW 

External  - 
HAL  ASWC 
reported  that 
HAL 

conducted 
urgent 
torpedo 
attack  on 

Poss  sub 

2275 

(classified 
poss  sub  low 

2) 

10:00:34 

actual  firing  and 
proper  external  report 
by  HAL  ASWC  (no 
consideration  for  fact 
that  ASW  weapons 
are  tight) 

sub-surface 

contact 

assess  as 
possible  sub 
and 

identified  as 
Track2275; 
HAL 

launches 
torpedo  at  it 
to  see  if  it 
moves  like  a 
sub 
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ORO 

SWC 

ARRO 

CANEWS 

ASWC 

Time 

Events 

Comments 

Send 

Receive 

Send 

Receive 

Send 

Receive 

Send 

Receive 

Send 

Receive 

10:01:28 

ASWC 
properly 
announcing 
torpedo  fire. 
Does  not 
consider  that 
"weapons 
tight”  (can  only 
fire  on  contact 
positively  ID'd 
as  hostile) 

10:02:42 

HAL  reassessed  2275 
as  poss  sub  low  2 

10:15:14 

new  underwater 
contact  Track2463 
appears  on  Link  from 
REG 

sub-surface 
Track2275 
classified  as 
poss  sub  low 

2 

10:15:17 

new 

underwater 
contact 
picked  up  by 
REG  and  put 
on  Link 

10:15:20 

ASW 

External  - 
ASWC  asked 
RT2  about 

2463 

10:16:05 

REG  reported 
conducted  urgent 
torpedo  attack  on 
track  2463,  assessed 
Akula  class  sub 

REG  assess 
Track2463 
as  Akula 
class  sub 

10:16:20 

D9L  (HAL  CH124-B) 
on  top  Hostile  2275 
and  didn't  report 
Madman 

HAL  helicopter 
reported  that 
they  didn't  see 
metal  when 
they  flew  over 
hostile 
Track2275 

10:16:30 

Track2463  made 

Hostile  by  REG; 
classified  as  Poss  sub 
low  1 

REG 

assessed 
Track2463 
as  hostile 
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ORO 

SWC 

ARRO 

CANEWS 

ASWC 

Time 

Events 

Comments 

Send 

Receive 

Send 

Receive 

Send 

Receive 

Send 

Receive 

Send 

Receive 

and 

classified  it 
as  poss  sub 
low  1 

10:16:45 

REG  holding  blade 
(hear  props)  on 

Hostile  2463 

REG  reports 
that  hear 
props  from 
hostile 
Track2463, 
confirming 
it's  a  sub 

10:16:49 

torpedo  bearing  line 
2463  appeared  on 

RT2  screen 
(detected  by  REG  & 
sent  on  link) 

REG  inputs 
torpedo 
bearing  line 
and  sends 
on  Link 

10:17:10 

ASWC  gives 

ORO 

situation 

report  on 

status  of 

Track2463 

and 

Track2275 

C&C - ORO  & 
ASWC  sitrep 
regarding  2463 
and  2275 

10:18:18 

REG  reassessed 
Track2463  from  Poss 
sub  low  1  to  Prob  sub 
based  on  torpedo 

REG  more 

confident 

that 

Track2463  is 
a  sub  based 
on  its  actions 
following 
torpedo  fire 

10:18:20 

new  surface 

Track7610  appears 
on  SWC  screen 

10:18:30 

HAL  helo  reported 
Madman  when  on  top 
hostile  2467 
(•formerly  2463  -  b/w 
10:17:14  &  10:18:45 
track  #  changed) 

HAL 

helicopter 
reported  that 
they  saw 
metal  when 
flying  over 
Track2467 
(formerly 

2463) 

10:18:55 

HAL  assessed  hostile 
2275  as  decoy; 

HAL 

concluded 
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ORO 

SWC 

ARRO 

CANEWS 

ASWC 

Time 

Events 

Comments 

Send 

Receive 

Send 

Receive 

Send 

Receive 

Send 

Receive 

Send 

Receive 

"correlated  2467  with 
2275";  taken  2275  off 
link 

Track2275  is 
a  decoy  and 
Track2467  is 
the  real 
hostile  sub 

10:18:55 

recommendation  to 
make  7610  suspect 
poss  Delhi  based  on 
proximity  to  downed 
aircraft  (CAP  sent  to 
shoot  down  Mainstay) 
and  proximity  to 
unassociated  ESM 
(7600) 

recommend 
Track7610 
is  possible 
Delhi  (Indian 
warship) 
based  on 
proximity 
relative  to 
Mainstay 
aircraft  shot 
down 

(Track2400) 

10:19:00 

surface  track  7610 
made  suspect 

10:19:24 

track  2467  reassigned 
as  2275;  in  place  of 
former  2275  put  in 
new  hostile  non-sub 
2310 

Poor 

picture/track 
management 
by  HAL  (and 
the  ASW 
Commander 
who  is 
control  staff 
so  not 
surprising 
here) 

10:19:45 

RT2  amplified  decoy 

10:23:50 

MPA  associated 
racket  7600  with 
suspect  7610 

10:24:30 

Recommend  make 

7610  Suspect  based 
on  proximity  to  SAG 
AOP 

recommended 
that  Track7610 
be  made 
suspect  based 
on  proximity  to 
downed 
aircraft 

10:25:20 

REG:  make  7610 

Hostile  based  on 
assoc  with  ESM,  prox 
to  AOP  for  downed 
a/c  and  behaviour. 

REG  makes 
Track7610 
hostile  based 
on  ESM  info 
and 
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ORO 

SWC 

ARRO 

CANEWS 

ASWC 

Time 

Events 

Comments 

Send 

Receive 

Send 

Receive 

Send 

Receive 

Send 

Receive 

Send 

Receive 

VAN  (control  staff) 
agreed  with  REG 

proximity  to 

downed 

aircraft 

10:26 

VAN  rec  to  AS  - 
Hostile  7610  outside 
weapons  range 
therefore  no  warnings 
to  be  issued  by  MPA 

VAN  states 
that 

Track7610  is 
outside 
weapons 
range  so  no 
warnings  will 
be  used  by 
Maritime 

Patrol 

Aircraft 
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Annex  D:  Elements  of  an  Independent  SME 
Pool  of  Naval  Retirees  for  the  COMDAT  Project 


This  Annex  identifies  the  recommended  composition  of  an  independent  SME  pool,  if  one  were  to  be 
considered,  using  the  example  of  the  COMDAT  project.  The  approach  taken  was  to  identify  the 
relevant  operational  functions  within  the  operations  sub-teams  that  would  apply  to  the  project,  and  then 
determine  whether  that  function  could  be  simulated  by  a  generic  naval  SME  with  verbal  input  to  the 
team,  or  would  require  a  qualified  operator  to  fulfil  the  requirements  of  the  function.  The  SME  pool 
would  be  comprised  of  representatives  from  the  positions  that  were  required  to  support  data  collection 
scenarios  and  whose  functions  could  not  be  simulated,  as  well  as  the  generic  naval  SME. 

The  functions  include: 

•  Radar  tracking/above  water  picture  compilation 

•  Underwater  picture  compilation 

•  Electronic  support  measures  (ESM) 

•  Fire  control 

•  Aircraft  control 

•  Warfare  sub-team  direction 

•  Ops  room  direction 

•  Command 


Function:  Radar  Tracking/Above  Water  Picture  Compilation 

•  Related  positions: 

-  Track  Supervisor  (TS). 

-  Air  Raid  Reporting  Operator  (ARRO). 

•  MOC  /  qualification  /  rank  /  experience: 

MOC:  Naval  Combat  Information  Operator  (NCIOP).  The  TS  is  normally  a  QL5.  The 
ARRO  has  a  billeted  position  of  a  QL3  but  is  often  a  QL4  or  QL5  in  practise.  Expected 
ranks  would  range  from  Ordinary  Seaman  (OS)  to  Master  Seaman  (MS),  with  one  to  eight 
years  experience. 

•  Responsibilities: 

Radar  detection  and  tracking,  above  water  picture  compilation  including  surface  and  air, 
physical  control  of  the  SPS  49  air  search  radar  and  SG  150  multi-purpose  radar,  and  link 
management. 

•  Criticality: 

-  TS  is  a  critical  role  in  the  ops  team  and  required  in  nearly  every  conceivable  scenario. 

-  ARRO  is  a  critical  role  in  the  Anti  Air  Warfare  (AAW)  sub-team  and  required  for  any 
AAW  scenario. 

-  Very  difficult  if  not  impossible  to  simulate  the  functions  of  these  two  operators. 
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•  Availability 

Experienced  SMEs  to  fill  this  position  are  likely  fairly  readily  available  from  within  navy 
resources.  Retired  SMEs  are  likely  to  be  available  in  the  HFX  area,  but  if  they  have  a 
second  career  it  may  be  difficult  to  ensure  their  availability  on  an  ad  hoc  basis. 

Under  Water  Picture  Compilation 

•  Related  positions: 

-  Hull  mounted  sonar  operator  (HMS  Operator) 

-  Towed  array  sonar  operators  (CANTASS  Operators) 

-  Sonobuoy  processing  system  operator  (SPS  Operator) 

-  Sonar  Control  Supervisor  (SCS) 

-  Anti-submarine  Plotting  Operator  (ASPO) 

•  MOC  /  qualification  /  rank  /  experience: 

MOC:  Tactical  Acoustic  Sensor  Operator  (TASOP),  except  the  ASPO  who  is  NCIOP.  The 
SCS  is  a  QL6A.  Other  operators  are  QL3-5.  The  ASPO  has  a  billeted  position  of  a  QL3 
but  is  often  a  QL4  or  QL5  in  practise.  Expected  ranks  would  range  from  Ordinary  Seaman 
(OS)  to  Petty  Officer  Second  Class  (P02),  with  one  to  14  years  experience. 

•  Responsibilities: 

Underwater  detection,  tracking  and  localization,  and  input  to  the  under  water  picture 
compilation  process. 

•  Criticality: 

Most  of  the  operator  functions  related  to  underwater  detection  are  transparent  and  could 
easily  be  simulated.  Even  the  role  of  the  SCS  could  be  simulated.  However,  the  ASPO  is  a 
critical  component  of  the  ASW  sub-team  and  would  be  required  in  any  ASW  scenario. 

•  Availability 

Experienced  SMEs  to  fill  this  position  are  likely  fairly  readily  available  from  within  navy 
resources.  Retired  SMEs  are  likely  to  be  available  in  the  HFX  area,  but  if  they  have  a 
second  career  it  may  be  difficult  to  ensure  their  availability  on  an  ad  hoc  basis 

Electronic  Support  Measures  (ESM) 

•  Related  positions: 

-  Communications  intercept  operator  (CIO  Operator) 

-  Canadian  Electronic  Warfare  System  operator  (CANEWS  Operator) 

-  Electronic  Warfare  Supervisor  (EWS) 

•  MOC  /  qualification  /  rank  /  experience: 

MOC:  Naval  Electronic  Sensor  Operator  (NESOP).  The  EWS  is  a  QL6A.  Other  operators 
are  QL3-5.  Expected  ranks  would  range  from  Ordinary  Seaman  (OS)  to  Petty  Officer 
Second  Class  (P02),  with  one  to  14  years  experience. 

•  Responsibilities: 

Detection  and  assessment  of  voice  and  radar  emissions,  and  input  to  the  above  water 
picture  compilation  process.  The  EWS  also  controls  soft  kill  measures  during  ASMD. 
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•  Criticality: 

These  functions  are  all  important  but  input  to  the  picture  compilation  process  would  be 
sporadic.  The  ASMD  function  is  critical  in  real  life  but  largely  transparent  during  team 
training.  It  would  be  reasonably  easy  for  a  non-NESOP  naval  SME  to  simulate  the  ESM 
function  at  a  basic  level  concurrent  with  other  functions. 

•  Availability 

Experienced  SMEs  to  fill  this  position  are  likely  fairly  readily  available  from  within  navy 
resources.  Retired  SMEs  are  likely  to  be  available  in  the  HFX  area,  but  if  they  have  a 
second  career  it  may  be  difficult  to  ensure  their  availability  on  an  ad  hoc  basis 

Fire  Control  (FC) 

•  Related  positions: 

-  Fire  control  operators  (FCO) 

-  Fire  Control  Supervisor  (FCS) 

•  MOC  /  qualification  /  rank  /  experience: 

MOC:  Naval  Electronic  Sensor  Operator  (NESOP).  The  FCS  is  a  QL5.  The  two  operators 
are  QL3-5.  Expected  ranks  would  range  from  Ordinary  Seaman  (OS)  to  Master  Seaman 
(MS),  with  one  to  1 0  years  experience. 

•  Responsibilities: 

Monitoring  and  reporting  the  status  of  the  fire  control  radars  and  fire  control  solution 
throughout  the  designation  to  engage  sequence.  They  may  report  the  visual  identification 
of  an  acquired  target  via  the  STIR  cameras.  They  also  control  aspects  of  the  57  MM  gun 
during  an  engagement. 

•  Criticality: 

Much  of  the  fire  control  function  is  redundant  and  can  be  performed  by  the  SWC 
independently  from  the  SWC  console.  The  vast  majority  of  verbal  reports  could  be 
omitted  during  a  simulation  without  seriously  detracting  from  realism.  Inputs  such  as  the 
visual  identity  of  a  target  could  easily  be  simulated  by  anyone  with  or  without  naval 
experience. 

•  Availability 

Experienced  SMEs  to  fill  this  position  are  likely  fairly  readily  available  from  within  navy 
resources.  Retired  SMEs  are  likely  to  be  available  in  the  HFX  area,  but  if  they  have  a 
second  career  it  may  be  difficult  to  ensure  their  availability  on  an  ad  hoc  basis. 

Aircraft  Control 

•  Related  positions: 

-  Shipbome  air  controller  (SAC) 

•  MOC  /  qualification  /  rank  /  experience: 

MOC:  Naval  Combat  Information  Operator  (NCIOP)  or  Maritime  Surface/Sub-surface 
Officer  (MARS).  The  NCIOP  is  a  QL5.  The  MARS  officer  is  a  “D”  level  qualified  officer. 
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Expected  ranks  would  be  Leading/Master  Seaman  (LS/MS)  or  Sub-Lieutenant/Lieutenant 
(N).  Each  would  have  four  to  six  years  experience  in  the  Navy. 

•  Responsibilities: 

Providing  the  interface  between  the  ship,  and  helicopters  and  maritime  patrol  aircraft 

•  Criticality: 

The  input  they  would  provide  to  the  picture  compilation  process  would  be  relatively 
sporadic  and  could  be  simulated  at  a  basic  level  by  a  non-SAC  qualified  naval  SME. 

•  Availability 

Experienced  SMEs  to  fill  this  position  are  likely  fairly  readily  available  from  within  navy 
resources.  Retired  SMEs  are  likely  to  be  available  in  the  HFX  area,  but  if  they  have  a 
second  career  it  may  be  difficult  to  ensure  their  availability  on  an  ad  hoc  basis. 

Warfare  Sub-team  Direction 

•  Related  positions: 

-  Above  Water  Weapons  Director  (AWWD) 

-  Anti-submarine  Warfare  Director  (ASWD) 

•  MOC  /  qualification  /  rank  /  experience: 

MOC:  NESOP,  NACOP  or  MARS.  The  NESOP  and  NACOP  are  QL6B,  and  the  MARS 
officer  is  a  “D”  level  qualified  officer.  Expected  ranks  would  be  POl  for  the 
NESOP/NACOP,  and  Lt(N)  for  the  officer.  The  NESOP/NACOP  would  have  12-15 
years  experience,  and  the  officer  five  to  eight  years. 

•  Responsibilities: 

Sub-team  supervision,  picture  compilation,  threat  assessment  and  weapons  engagement  of 
contacts  in  their  particular  warfare  area 

•  Criticality: 

Both  positions  are  critical  for  scenarios  dealing  with  their  domain,  and  they  could  not  be 
simulated. 

•  Availability 

Experienced  SMEs  to  fill  this  position  are  likely  fairly  readily  available  from  within  navy 
resources.  Few  SACs  would  retire  from  this  position;  more  likely  they  carry  on  in  their 
career  and  are  somewhat  dated  by  the  time  they  retire.  If  they  did  retire  from  these 
positions  they  would  likely  have  a  second  career  it  may  be  difficult  to  ensure  their 
availability  on  an  ad  hoc  basis. 

Ops  Room  Direction 

•  Related  positions: 

-  Operations  Room  Officer  (ORO) 

-  Operations  Room  Supervisor  (ORS) 
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•  MOC  /  qualification  /  rank  /  experience: 

MOC:  The  ORO  is  MARS,  ORO  Course  qualified.  Expected  ranks  would  be  senior  Lt(N) 
or  junior  LCdr,  with  eight  to  12  years  experience.  The  ORS  is  NCIOP,  QL6A  qualified. 
Expected  rank  would  be  P02,  with  eight  to  12  years  experience. 

•  Responsibilities: 

The  ORO  is  overall  responsible  for  all  activities  within  the  ops  room,  which  includes 
supervising  and  directing  all  aspects  of  picture  compilation,  threat  assessment  and 
weapons  engagement.  The  directors  and  their  teams  are  responsible  to  the  ORO.  The 
ORO  provides  the  interface  to  the  ship’s  captain.  The  ORS  provides  general  supervision 
and  coordination  within  the  ops  room,  and  interfaces  with  the  Communications  Control 
Room  for  communications  needs. 

•  Criticality: 

The  ORO  is  critical  regardless  of  the  scenario  and  could  not  be  simulated.  Although  the 
ORS  provides  an  important  function  at  sea,  it  is  largely  in  the  background  and  this 
position  is  not  critical  to  the  picture  compilation  process. 

•  Availability 

Experienced  SMEs  to  fill  this  position  are  likely  fairly  readily  available  from  within  navy 
resources.  Few  OROs  and  ORSs  retire  from  this  position;  more  likely  they  carry  on  in 
their  career  and  are  somewhat  dated  by  the  time  they  retire.  If  they  did  retire  from  these 
positions  they  would  likely  have  a  second  career  it  may  be  difficult  to  ensure  their 
availability  on  an  ad  hoc  basis. 

Command 

•  Related  positions: 

-  Commanding  Officer  (CO) 

-  Executive  Officer  (XO) 

•  MOC  /  qualification  /  rank  /  experience: 

MOC:  MARS,  command  qualified.  Expected  ranks  would  be  senior  LCdr  to  Capt(N), 
with  14-30  years  experience. 

•  Responsibilities: 

Final  supervision  and  authority  of  all  activities  of  the  ship.  Sets  or  approves  operational 
priorities.  In  a  multi-threat  situation,  may  directly  participate  in  one  threat  area, 
delegating  the  other  area  to  the  ORO  while  demanding  timely  updates. 

•  Criticality: 

For  complete  reality,  and  particularly  if  assessing  ORO  roles,  a  command  presence  would 
be  critical  during  scenarios  involving  engagements.  For  picture  compilation  processes, 
the  role  of  command  could  be  simulated  by  a  suitable  SME. 

•  Availability 

Experienced  SMEs  to  fill  this  position  are  likely  fairly  readily  available  from  within  navy 
resources.  Retired  SMEs  are  likely  to  be  available  in  the  HFX  area,  but  if  they  have  a 
second  career  it  may  be  difficult  to  ensure  their  availability  on  an  ad  hoc  basis.  It  would 
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probably  be  easier  to  find  retired  COs  willing  to  participate  than  some  of  the  other 
positions. 

Tabular  Summary 

The  following  table  summarizes  the  recommended  composition  of  an  independent  naval  SME  pool. 
The  table  indicates  which  elements  could  be  simulated  by  a  suitable  generic  naval  SME  who  may 
fulfil  the  requirements  of  multiple  elements  simultaneously,  and  those  elements  that  due  to  the  nature 
of  their  required  expertise  or  the  scope  of  their  role  within  the  scenario,  would  have  to  be  represented 
by  SMEs  with  qualifications  and  experience  in  those  positions. 


Function 

Required  elements 

Critical 

Core/support 

Pool  element 

Radar  tracking/  AWW  picture  compilation 

Track  Sup 

Y 

Core 

QL5/6A  NCIOP 

ARRO 

Y* 

Core 

QL5  NCIOP 

Under  water  picture  compilation 

SCS 

N 

Support 

Generic  SME 

ASPO 

Y* 

Core 

QL5  NCIOP 

Electronic  support  measures 

EWS 

N 

Support 

Generic  SME 

Fire  Control 

FCO 

N 

Support 

Generic  SME 

Aircraft  control 

SAC 

N 

Support 

Generic  SME 

Warfare  sub-team  direction 

AWWD 

Y* 

Core 

QL6B  NESOP 

Lt(N)  director 

ASWD 

Y* 

Core 

QL6B TASOP 

Lt(N)  director 

Ops  room  direction 

ORS 

N 

Support 

Generic  SME 

ORO 

Y 

Core 

ORO 

Command 

OOW 

N 

Support 

Generic  SME 

CO/XO 

N 

Support 

Generic  SME 

*  Dependent  on  focus  of  scenario 


A  core  SME  pool  would  consist  of  an  ORO,  an  AWWD  (either  officer  or  NESOP),  an  ASWD  (either 
officer  or  TASOP),  a  QL6A  NCIOP,  and  one  or  two  QL5  NCIOPs.  In  addition,  at  least  one  ORO 
would  be  required  to  fulfil  the  higher  level  requirements  of  a  generic  SME,  and  another  SME, 
preferably  an  officer  director,  would  be  required  to  fulfil  the  other  requirements.  The  pool  would  have 
to  consist  of  multiple  people  for  each  element  for  scheduling  redundancy.  If  a  pool  was  intended  to 
fulfil  the  requirements  of  multiple  scenario  runs  with  new  players,  then  obviously  the  duplication  of 
elements  increases  proportionately. 
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14.  ABSTRACT 


(U)  This  report  compiles  the  information  contained  in  three  Technical  Memoranda  pertaining  to  the 
development  of  Measures  of  Performance  (MOP)  for  Multi-Sensor  Data  Fusion  technology  (MSDF),  and 
evaluates  logistics  and  the  utility  of  the  Operations  Room  Team  Trainer  (ORTT)  to  gather  these  MOPs. 
Each  Technical  Memorandum  is  preceded  by  a  summary  of  its  content.  This  report  recounts  the 
development  of  a  strategy  for  collection  of  MOPs,  the  conduct  of  a  proof-of-concept  trial  at  ORTT  (with 
attendant  identification  of  potential  improvements  to  the  data  collection  strategy),  and  the  investigation 
of  methods  to  reduce  the  logistical  and  availability  problems  associated  with  obtaining  access  to  Subject 
Matter  Experts  (SME).  As  a  result  of  this  work,  a  further,  full-scale,  investigation  of  the  archived  ORTT 
perfonnance  data  to  provide  performance  data  for  the  COMDAT  Technology  Demonstration  Programme 
was  recommended. 

(El)  Le  present  rapport  comprend  une  compilation  de  l’infonnation  contenue  dans  trois  documents 
techniques  portant  sur  l’elaboration  de  mesures  du  rendement  de  la  technologie  de  la  fusion  de  donnees 
multicapteurs  (FDMC),  ainsi  qu'une  evaluation  de  la  logistique  et  de  l’utilite  du  simulateur  pour  l’equipe 
de  la  salle  des  operations  (SESO)  en  vue  de  la  collecte  des  mesures  du  rendement.  Chaque  document 
technique  est  precede  d’un  resume  de  son  contenu.  Le  present  rapport  recapitule  felaboration  d’une 
strategic  de  collecte  des  mesures  du  rendement,  la  conduite  de  l'essai  de  validation  de  principe  au  SESO 
(l’operateur  identifiant  les  ameliorations  possibles  a  apporter  a  la  strategic  de  collecte  des  donnees)  et 
l'examen  des  methodes  visant  une  reduction  des  problemes  de  logistique  et  de  disponibilite  associes  a 
faeces  aux  experts  en  la  matiere  (EM).  Suite  a  ce  travail,  un  autre  examen  pleine  echelle  des  donnees 
archivees  sur  le  rendement  du  SESO  a  ete  recommande  en  vue  de  la  fourniture  de  donnees  sur  le 
rendement  dans  le  cadre  du  programme  de  demonstration  de  la  technologie  d’aide  aux  decisions  de 
commandement  (COMDAT). 
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